OpenFront Harness

A deterministic, inspectable harness in which one LLM plays a fixed OpenFront match on Japan against three built-in medium-difficulty nations. Every run produces a complete OpenFront replay plus a decision-by-decision trace of observations, legal actions, selected intents, latency, tokens, cost, hashes, and outcome.

The harness is a separate project. OpenFrontIO/ stays at the pristine v0.32.9 tag and is treated as a read-only source dependency; the harness imports its core and builds its renderer without patching tracked game files.

This is a portfolio project and a first step toward a reproducible strategy-agent benchmark. The scenario is deliberately immutable: changing it creates a new benchmark version.

What you can inspect

• A real OpenFront v0.32.9 simulation, not a simplified game clone
• One LLM spawned in Kanto against Hokkaido, Shikoku, and Kansai
• A bounded menu of currently legal core intents every 100 ticks
• Exactly two action slots per model decision, including explicit holds
• Structured model output validated against both JSON Schema and the legal menu
• Full intent-based replay in OpenFront's existing renderer
• A synchronized trace showing the public strategy note, actions, state, total latency, per-attempt TTFT, generation time, TPOT, tokens, provider, and cost
• Gzipped, versioned run artifacts suitable for later benchmark ingestion
• A bundled real-model sample that works without an API key

The exact scenario is documented in design-decision.md, and the project retrospective is in writeup.md.

Quick start

Requirements: Node.js 24 and npm.

git clone --recurse-submodules https://github.com/fahimahmedx/openfront-harness.git
cd openfront-harness
npm run inst
cp example.env .env
# Add OPENROUTER_API_KEY to .env
npm run build
RATE_LIMIT_SALT=local-development npm start

For an existing clone, fetch the pinned OpenFront source with git submodule update --init --recursive before installing dependencies.

Open http://localhost:3000. The server reads the workspace-level .env; no environment file is added to the OpenFront checkout.

Commands

npm test               # focused unit and artifact tests
npm run verify:sample  # replay recorded actions and verify tick/hash/outcome
npm run build          # type-check and production Vite build
npm run sample         # spend live API credits to create a new sample
npm run refresh:sample # rebuild sample from its recorded trace, no API call

npm run sample uses the same hard $1 run cap as the public app. verify:sample is the stronger determinism check: it feeds the saved decisions back through OpenFront and requires the final winner, tick, and state hash to match.

Fixed benchmark preset

The model is never given permission to invent a command. The server enumerates a deterministic set of legal action IDs from current state and gives each named output slot its own strict enum. Both slots execute simultaneously, so the second action cannot depend on the first action's result. Repeatable troop actions may fill both slots because each commitment is already bounded to half of the shared safe budget; repeated non-repeatable actions normalize to a hold. Cooperative and hostile actions toward the same opponent are rejected as a conflicting pair and retried. The selected IDs resolve to ordinary OpenFront intents. Invalid output is retried once and then converted to two holds. Five consecutive complete decision failures abort a run.

Troop-spending actions share one two-slot budget instead of independently spending percentages of the same balance. Neutral expansion preserves 15% of troop capacity, ordinary combat preserves 35% and unlocks at 55% capacity, and attacks against stronger defenders are withheld. During an invasion, unrelated troop spending is suppressed and bounded counterattacks preserve a 15% emergency reserve.

Artifact shape

New .json.gz artifacts use schema version 2; schema-version-1 artifacts remain readable. Each artifact contains:

• scenario, source commit, model, provider, prompt version, and reasoning configuration;
• start/end timestamps and exact token/cost usage;
• winner, placement, terminal tick, simulated time, and final hash;
• every normalized observation and legal candidate menu;
• selected and applied action IDs, post-execution lifecycle results, fallback state, and request metrics;
• an OpenFront GameRecord containing the initial configuration and sparse turns.

Run files are written atomically. A small pending JSON checkpoint is updated after each decision so interrupted work is visible rather than silently disappearing.

HTTP surface

Railway deployment

The included railway.toml selects the multi-stage Dockerfile and /api/health. Configure:

• OPENROUTER_API_KEY — required for fresh public runs;
• RATE_LIMIT_SALT — required in production; use a long random value;
• PUBLIC_BASE_URL — the deployed HTTPS origin;
• SOURCE_URL — the public repository for this modified AGPL deployment;
• optionally OPENROUTER_MODEL, OPENROUTER_PROVIDER, and CDN_BASE.

Attach a Railway Volume at /data. The image sets RUN_DATA_DIR=/data; without a volume, generated artifacts and quota state disappear on redeploy. The bundled sample remains available because it is part of the image.

Public generation is intentionally conservative: one active run globally, one launch per hashed IP per UTC day, five launches per UTC day, a ten-minute wall-clock limit, and a $1 model-cost ceiling. Only an HMAC of the IP is persisted.

Repository layout

src/                          scenario, runner, agent, storage, API
src/client/                   dashboard, replay bootstrap, trace panel
tests/                        focused deterministic tests
data/                         local run artifacts grouped by run folder
resources/harness/            bundled real-model replay artifacts
OpenFrontIO/                   untouched v0.32.9 source dependency
vite.config.ts                external build and in-memory replay adapter
design-decision.md            decisions with pros and cons
writeup.md                    publishable project retrospective

Upstream and license

This project consumes an unmodified checkout of OpenFrontIO/OpenFrontIO at tag v0.32.9. OpenFront is an online real-time strategy game focused on territorial control and alliance building and is itself a fork/rewrite of WarFront.io.

OpenFront source code is licensed under the GNU Affero General Public License v3.0, and this harness is distributed under compatible AGPL terms. Assets retain their applicable CC BY-SA 4.0 terms. Network deployments must offer corresponding source and preserve the visible notices. See LICENSE, LICENSE-ASSETS, and LICENSING.md.

© OpenFront and Contributors