From a running game to a diagnosed leaderboard
Three stages — design, replication, evaluation — run under two controlled axes (model / harness), and every failed instance is traced to a phase and sub-mode.
GameReplica is a benchmark for black-box visual game replication. An agent sees a running web game only through screenshots and an action interface — no source, no docs — must induce its rules by playing, and rebuild it as a single self-contained, verifiable HTML page.
Unlike SWE / UI-generation benchmarks that hand the agent an issue, mockup or tutorial, GameReplica gives no specification at all. The rules live implicitly inside a running system; the agent must uncover them through observation and active interaction, then reproduce them as code.
Each base game is expanded into 5 levels (L1–L5) along explicit rule- and state-complexity axes, each with a semantically aligned spec, rubric and hidden source.
The agent's only channel is action → screenshot (click / key / reset). It probes, infers state transitions and win/lose rules, and writes one runnable HTML plus an induced spec doc.
The replica is scored on semantic consistency (Rubric), source-code structure (Code) and visual fidelity (Vision) — never touching the original's implementation.
The suite spans rule-driven puzzles and two-player board games, chosen so faithful replication demands recovering genuine mechanics — not just matching a static screenshot.
Three stages — design, replication, evaluation — run under two controlled axes (model / harness), and every failed instance is traced to a phase and sub-mode.
For every system, visual fidelity far exceeds rule fidelity (Claude 93.1 > 67.4 > 54.2). Agents readily produce pages that look similar but fail to reproduce the actual game behavior.
Swapping the model spans an 18× Code-score range; swapping the harness only 1.26×. A strong model sets the ceiling; the harness only changes how the task is executed.
From L1 to L5 Claude stays nearly flat (72.1→68.6) while Grok collapses (35.9→15.5) and GLM (25.2→11.0). Capability shows up as robustness at the hardest levels.
Rubric / Code are tier-weighted means; Vision is the screenshot-comparison score; Overall is their equal-weight mean (×100, higher = closer to original). Click a column to sort.
Tokens (M) and Time (s) are averaged over successfully completed instances only. Done = instances (of 125) that stably produce a scorable page.
Each point aggregates 25 games at that level. Weaker systems collapse as difficulty rises; the strongest stays flat.
Pick a game, a level and the systems to compare. Every frame is a live, runnable page — the original is pinned first. Click "Open" for full screen.
Across 1,250 instances every outcome is placed in a five-tier taxonomy (F1–F5, bottom-up, first match wins); each failure is then traced to a phase and a concrete sub-mode.
Share of all attributed failures by phase.
Most frequent concrete error patterns.
