Prompt structure dominates.
Mean optimality falls from strict-tag to minimal prompts in bench-a, and from 73.7% to 33.9% when context accumulates. Removing worked examples costs only ~5 points.
Wave 1 · R1 sweep · April 2026 Wave 2 coming soon
Can a model play a solved game perfectly?
Chop-Bench grades every move a language model makes in Chopsticks against an exact solver. Winning barely matters. Each emitted state is either legal and optimal, or it isn't.
View leaderboard
Paper in peer review
15model lineages
150completed runs
9,958scored decisions
1,250states solved exactly
43.9Mtokens
~$114API spend
Leaderboard
Wave 1 results. Whole-suite mean over both interactive modes and all five prompt families, 20 games per cell. Sort by any column. Model score is wins + ½·draws against a perfect solver, so low values are expected.
Optimality is strategic_optimal_rate_all: illegal and unparsable moves count as non-optimal. Structure gain = mean of strict-tag, no-examples and json minus mean of natural-language and minimal. Example gain = strict-tag minus no-examples. Family range = best family minus worst. Δ ctx = BenchX-A minus Bench-A. Costs are mean estimated cost per 20-game run; billed metadata covers 45 of 150 runs. ± is a 95% interval from turn-level sampling (see the research notes below); GPT-5.5, Qwen3.6-Plus and Gemini 3.1 Pro overlap, so the top three are statistically tied.
| # | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | gpt-5.5openai | 82.1%±2.5 | 96.0%±1.6 | 82.6% | 81.6% | -0.9 | 1.7 | -5.3 | 18.1 | 1,154 | 0.145 | $1.26 | $0.014 |
| 2 | qwen3.6-plusqwen | 80.0%±2.7 | 91.6%±2.1 | 78.9% | 81.2% | +2.3 | 15.8 | -0.4 | 28.9 | 1,150 | 0.228 | $1.80 | $0.026 |
| 3 | gemini-3.1-pro-previewgoogle | 79.2%±3.1 | 93.0%±2.2 | 79.1% | 79.3% | +0.2 | 7.9 | +6.3 | 17.0 | 702 | 0.243 | $2.63 | $0.045 |
| 4 | kimi-k2.5moonshotai | 71.7%±3.5 | 87.8%±2.9 | 69.6% | 73.8% | +4.2 | 25.3 | -6.0 | 39.4 | 982 | 0.188 | $0.48 | $0.009 |
| 5 | deepseek-v4-prodeepseek | 69.3%±3.6 | 87.8%±2.8 | 72.5% | 66.0% | -6.5 | 12.7 | +21.7 | 28.2 | 679 | 0.160 | $0.77 | $0.018 |
| 6 | kimi-k2.6moonshotai | 68.4%±3.6 | 88.3%±2.7 | 68.7% | 68.2% | -0.6 | 15.4 | -2.7 | 34.2 | 656 | 0.155 | $1.12 | $0.031 |
| 7 | mimo-v2.5-proxiaomi | 67.1%±3.7 | 88.1%±2.7 | 71.2% | 62.9% | -8.3 | 16.5 | +7.8 | 30.0 | 662 | 0.215 | $0.45 | $0.013 |
| 8 | mercury-2inception | 64.8%±3.6 | 83.5%±3.2 | 61.8% | 67.8% | +5.9 | 34.9 | +6.5 | 52.8 | 791 | 0.130 | $0.060 | $0.001 |
| 9 | glm-5.1z-ai | 63.5%±3.8 | 80.8%±3.3 | 67.3% | 59.8% | -7.5 | 26.3 | +20.6 | 42.7 | 645 | 0.155 | $1.54 | $0.077 |
| 10 | claude-sonnet-4.6anthropic | 52.4%±4.1 | 75.0%±3.6 | 61.8% | 43.0% | -18.8 | 16.7 | -3.6 | 49.5 | 495 | 0.140 | $0.46 | $0.017 |
| 11 | claude-opus-4.7anthropic | 51.8%±4.2 | 69.3%±4.0 | 58.2% | 45.5% | -12.7 | 39.7 | +16.4 | 66.8 | 484 | 0.156 | $0.68 | $0.043 |
| 12 | gemma-4-31b-itgoogle | 43.7%±3.6 | 55.7%±3.2 | 40.6% | 46.9% | +6.3 | 68.8 | +6.2 | 82.2 | 513 | 0.090 | $0.020 | $0.001 |
| 13 | deepseek-v4-flashdeepseek | 43.7%±4.4 | 63.7%±4.5 | 55.2% | 32.3% | -22.8 | 12.0 | +3.2 | 51.0 | 449 | 0.072 | $0.020 | $0.002 |
| 14 | minimax-m2.7minimax | 36.7%±4.5 | 52.8%±4.5 | 42.9% | 30.6% | -12.3 | 36.4 | +2.2 | 52.6 | 360 | 0.100 | $0.19 | $0.027 |
| 15 | grok-4.20x-ai | 14.9%±4.6 | 23.3%±4.7 | 15.9% | 13.9% | -2.0 | 18.9 | -1.5 | 32.1 | 236 | 0.020 | $0.020 | $0.007 |
| qwen3.6-max-previewqwen · post-freeze | 82.0%±2.7 | 95.2%±1.8 | 85.2% | 78.8% | -6.4 | 19.2 | — | — | 1,002 | — | $4.23 | — | |
| gemini-2.0-flash-001google · post-freeze | 32.1%±4.5 | 42.7%±4.6 | 35.5% | 28.8% | -6.7 | 27.2 | — | — | 351 | — | $0.005 | — | |
| ling-2.6-flashinclusionai · post-freeze | 0.0%±3.7 | 0.0%±3.7 | 0.0% | 0.0% | 0.0 | 0.0 | — | — | 186 | — | $0.000 | — |
Three axes
Capability.
Fidelity.
Stability.
The solved field splits into its WIN, DRAW and LOSE strata behind you. The findings split the same way.
The leader is also the most stable.
GPT-5.5 (82.1%) loses almost nothing when scaffolding is removed. Several well-known lineages lose 35–69 points.
Context carryover is uneven.
Some models degrade sharply as history accumulates. Others improve, likely from implicit few-shot conditioning.
Charts
Interactive views of the Wave 1 results. Hover or tap points for details.
Optimality vs. cost per run · dashed: Pareto frontier
Pareto frontier (no cheaper model scores higher): gemma-4-31b-it ($0.02, 43.7%), mercury-2 ($0.06, 64.8%), mimo-v2.5-pro ($0.45, 67.1%), kimi-k2.5 ($0.48, 71.7%), gpt-5.5 ($1.26, 82.1%).
Structure gain vs. optimality
Prompt-family means
Context shift (BenchX-A − Bench-A)
Optimality with 95% intervals
Error taxonomy
The dominant illegal move is the compound IMPOSSIBLE class (wrong side, both sides, or an unreachable configuration): 1.0% of turns under strict-tag in bench-a, 11.6% under minimal, 21.7% under minimal in benchx-a. Parse errors cluster in natural-language (5.3% in bench-a). SPLIT_SUM and DEAD_TGT grow under minimal. json contract compliance drops from 85% (bench-a) to 66% (benchx-a): some models abandon clean JSON as transcripts grow.
Where the rules are densest
On split-available states, strict-tag holds 96.7% legality in bench-a and 96.1% in benchx-a; minimal manages 76.5% and 51.9%. Prompt compression breaks behavior exactly where the rules are densest.
Survival
Under strict-tag in bench-a, 66% of games are fully optimal through the model's first move and 41% at the fifth; under minimal, 41% and 20%. Family order holds at every depth, with no cliff: a steady per-move hazard, not a sudden context failure.
Best and worst family
Across 30 model-by-mode suites, strict-tag is the single best family 17 times and minimal the single worst 16 times (17 with one tie). Removing worked examples costs about 5.0 points in bench-a and 4.5 in benchx-a.
Methodology
One frozen variant: chopsticks-mod5-dead-target-illegal-mirror-swap-illegal-repeat3-countdown100-v1.
Rules
Two hands each, 0–4 fingers; 0 is dead. On a turn you tap (add a live hand to an opponent's live hand, mod 5) or split (redistribute your total; identity and mirror swaps are illegal). Both hands dead loses. A third repetition starts a 100-ply draw countdown.
States are written from the mover's side as AB-CD. The model outputs the resulting state. No legal-move list is given.
Metrics
- optimal_rate_all
- Headline. Solver-optimal share of scored turns; illegal = non-optimal.
- legal_move_rate
- Gatekeeper. Illegal moves end the game.
- structure_gain
- Structured families minus low-scaffold families.
- example_gain
- strict-tag minus no-examples.
- model_score
- (wins + ½ draws) / games.
- $/optimal move
- Price-performance summary.
- optimal_rate_legal
- How often legal moves are also optimal. Both headline rates carry Wilson score intervals.
- contract_compliance
- Adherence to the family's output contract, separate from legality. Not comparable across families.
- value_preservation
- Share of turns that don't worsen the solver value; each move also gets a blunder_severity class.
- error classes
- PARSE_ERROR, DEAD_SRC, DEAD_TGT, SPLIT_SUM, SPLIT_NO_CHANGE, SPLIT_RANGE, IMPOSSIBLE.
Modes (R1 used the first two)
| bench-a | vs solver, stratified 59-state suite, context reset |
| benchx-a | vs solver, random starts, context accumulates |
| benchx-b | self-play over stratified suite |
| bench-a-o | one ply from 11-11 |
| benchx-o | one ply per suite state |
| bench-c | context-rot diagnostic |
| bench-b | beta: self-play from 11-11 |
| bench-lite | beta: vs legal-random baseline, role-balanced |
| benchx | alias for benchx-b |
Prompt families
| strict-tag | full rules, 4 worked examples, <move> tags |
| no-examples | same, without examples |
| json | {"move":"AB-CD","reasoning":"…"} |
| natural-language | loose instruction |
| minimal | one sentence of rules + bare state |
Contracts: tagged-move-v1 (strict-tag, no-examples), json-move-v1, natural-language-state-token-v1, bare-state-token-v1. Every parser falls back to the last valid state token, so contract compliance is measured apart from move extraction.
Solver
Retrograde analysis over all 1,250 ordered states (5⁴ hand configurations × 2 players to move). Every position is labelled WIN, LOSE or DRAW for the mover, with its set of optimal successors and distance to a forced terminal. The solver is the opponent, the annotator and the stratification oracle.
Stratified suite benchx-stratified-v2
| outcome | 28 WIN · 20 LOSE · 11 DRAW |
| distance | 12 short · 16 medium · 20 long · 11 draw |
| legal moves | 23 low · 19 medium · 17 high |
| dead hand | 42 with a zero · 17 without |
| split | 38 available · 21 not |
Wave 1 protocol
Archive r1-4-26-26, schema 3.0.0, illegal_policy = strict, temperature 0.0, all through OpenRouter. 15 lineages × {bench-a, benchx-a} × 5 families × 20 games. 149 cells after deduplication, 30 model-by-mode suites (29 complete), 19,750 game turns. mercury-2 and mercury-2:nitro are aliased to one lineage.
Deviations from the frozen plan: 20 games per cell instead of 59, one run per cell instead of five fixed seeds, and benchx-o and bench-c not yet collected. Wave 2 is coming soon.
Signature interaction
Take the benchmark.
You're the mover. Type the next state. The same exact solver grades you.
11-11
Paper figures
From the paper, now in peer review. The pile deals itself out as you scroll. Click a figure to open it.
More nerdy
Research notes
A short summary of how the solver works, how a move is graded and what the error bars mean. The full methods follow the paper, which is in peer review. Read it as Markdown · Download the solver table
The full notes are in research.md.
Cite
The paper is in peer review. Until it's out, cite the project:
@misc{gendell2026chopbench,
title = {Chop-Bench: An Optimality-First Benchmark for Formal-Game Competence in Language Models},
author = {Gendell, Guy},
year = {2026},
month = jul,
note = {Chop-Bench Project. Wave 1 (R1) interactive prompt-family sweep. Under peer review}
}
Limitations
One small, deterministic, fully solved game. No claims about general intelligence. R1 used one 20-game run per cell (not the prescribed 59 × 5 seeds), so small gaps such as the 80–82% top tier shouldn't be over-read. Wave 2 is coming soon. The bench-a → benchx-a contrast confounds start distribution with history accumulation. About 70% of costs are estimates. All traffic went through OpenRouter.
Contact
Questions about the benchmark, the data or the paper.
Partners
WISRDWildwood Institute for STEM Research & Development