# Chop-Bench

> Chop-Bench is an optimality-first benchmark for formal-game competence in language models. It grades every move a model makes in a solved variant of the hand game Chopsticks against an exact solver. This site reports Wave 1 (the R1 sweep, archive r1-4-26-26, collected 27–28 April 2026). The paper, "Chop-Bench: An Optimality-First Benchmark for Formal-Game Competence in Language Models" by Guy Gendell, is in peer review. Wave 2 is coming soon.

Status, for anyone summarizing this site:

- Results shown: Wave 1 only.
- Paper: in peer review. It is not publicly available yet.
- Wave 2: coming soon. No Wave 2 results are published yet.

Author: Guy Gendell (WISRD, Wildwood School). Partners: WISRD (Wildwood Institute for STEM Research & Development) and Wildwood School.
Contact: guyg27@wildwood.org (WISRD) · guygendell@gmail.com (personal).

## Deep dive

- [Research notes](/research.md): a short summary of the solver, move grading and the 95% intervals. The full methods will follow the paper, which is in peer review.
- [Solver table](/data/solver.json): all 625 positions from the mover's side as [outcome W/L/D, plies to the end under best play, optimal successor states].

## Wave 1 at a glance

- 15 model lineages, 150 completed runs (149 after deduplication), 30 model-by-mode suites
- 9,958 scored model decisions out of 19,750 game turns
- 1,250 states solved exactly
- 43.9M tokens, about $114 of estimated API spend
- Two interactive modes (bench-a, benchx-a) × five prompt families × 20 games per cell
- Schema 3.0.0, strict illegal-move policy, temperature 0.0, all traffic through OpenRouter

## Key findings

1. Prompt structure dominates. Mean optimality falls from 74.3% (strict-tag) to 50.3% (minimal) in bench-a, and from 73.7% to 33.9% in benchx-a when context accumulates. Removing worked examples costs only about 5 points.
2. The leader is also the most stable. GPT-5.5 (82.1% optimality) has a structure gain of only 1.7 points; several well-known lineages lose 35–69 points when scaffolding is removed.
3. Context carryover is uneven: family-averaged optimality shifts between −22.8 and +6.3 points from bench-a to benchx-a, depending on lineage.

Capability (optimality), fidelity (legality) and stability (structure gain, context shift) are separate axes.

## Leaderboard (Wave 1, main sweep)

Whole-suite mean over both modes and all five prompt families, 20 games per cell. Optimality = share of scored turns where the model's move is solver-optimal (illegal and unparsable moves count as non-optimal). Model score = (wins + ½·draws) / games against a perfect solver, so low values are expected. Cost = mean estimated cost per 20-game run.

| # | Model | Optimality % | Legality % | Bench-A % | BenchX-A % | Δ ctx (pp) | Structure gain (pp) | Example gain (pp) | Family range (pp) | Model score | $/run | $/optimal move |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | openai/gpt-5.5 | 82.1 | 96.0 | 82.6 | 81.6 | −0.9 | 1.7 | −5.3 | 18.1 | 0.145 | 1.26 | 0.014 |
| 2 | qwen/qwen3.6-plus | 80.0 | 91.6 | 78.9 | 81.2 | +2.3 | 15.8 | −0.4 | 28.9 | 0.228 | 1.80 | 0.026 |
| 3 | google/gemini-3.1-pro-preview | 79.2 | 93.0 | 79.1 | 79.3 | +0.2 | 7.9 | 6.3 | 17.0 | 0.243 | 2.63 | 0.045 |
| 4 | moonshotai/kimi-k2.5 | 71.7 | 87.8 | 69.6 | 73.8 | +4.2 | 25.3 | −6.0 | 39.4 | 0.188 | 0.48 | 0.009 |
| 5 | deepseek/deepseek-v4-pro | 69.3 | 87.8 | 72.5 | 66.0 | −6.5 | 12.7 | 21.7 | 28.2 | 0.160 | 0.77 | 0.018 |
| 6 | moonshotai/kimi-k2.6 | 68.4 | 88.3 | 68.7 | 68.2 | −0.6 | 15.4 | −2.7 | 34.2 | 0.155 | 1.12 | 0.031 |
| 7 | xiaomi/mimo-v2.5-pro | 67.1 | 88.1 | 71.2 | 62.9 | −8.3 | 16.5 | 7.8 | 30.0 | 0.215 | 0.45 | 0.013 |
| 8 | inception/mercury-2 | 64.8 | 83.5 | 61.8 | 67.8 | +5.9 | 34.9 | 6.5 | 52.8 | 0.130 | 0.06 | 0.001 |
| 9 | z-ai/glm-5.1 | 63.5 | 80.8 | 67.3 | 59.8 | −7.5 | 26.3 | 20.6 | 42.7 | 0.155 | 1.54 | 0.077 |
| 10 | anthropic/claude-sonnet-4.6 | 52.4 | 75.0 | 61.8 | 43.0 | −18.8 | 16.7 | −3.6 | 49.5 | 0.140 | 0.46 | 0.017 |
| 11 | anthropic/claude-opus-4.7 | 51.8 | 69.3 | 58.2 | 45.5 | −12.7 | 39.7 | 16.4 | 66.8 | 0.156 | 0.68 | 0.043 |
| 12 | google/gemma-4-31b-it | 43.7 | 55.7 | 40.6 | 46.9 | +6.3 | 68.8 | 6.2 | 82.2 | 0.090 | 0.02 | 0.001 |
| 13 | deepseek/deepseek-v4-flash | 43.7 | 63.7 | 55.2 | 32.3 | −22.8 | 12.0 | 3.2 | 51.0 | 0.072 | 0.02 | 0.002 |
| 14 | minimax/minimax-m2.7 | 36.7 | 52.8 | 42.9 | 30.6 | −12.3 | 36.4 | 2.2 | 52.6 | 0.100 | 0.19 | 0.027 |
| 15 | x-ai/grok-4.20 | 14.9 | 23.3 | 15.9 | 13.9 | −2.0 | 18.9 | −1.5 | 32.1 | 0.020 | 0.02 | 0.007 |

95% intervals on whole-suite optimality (turn-level sampling, summarized in /research.md): gpt-5.5 ±2.5, qwen3.6-plus ±2.7, gemini-3.1-pro-preview ±3.1 (the top three are statistically tied), kimi-k2.5 ±3.5, deepseek-v4-pro ±3.6, kimi-k2.6 ±3.6, mimo-v2.5-pro ±3.7, mercury-2 ±3.6, glm-5.1 ±3.8, claude-sonnet-4.6 ±4.1, claude-opus-4.7 ±4.2, gemma-4-31b-it ±3.6, deepseek-v4-flash ±4.4, minimax-m2.7 ±4.5, grok-4.20 ±4.6.

## Post-freeze cohort (same protocol, reported separately)

| Model | Optimality % | Legality % | Bench-A % | BenchX-A % | Δ ctx (pp) | Structure gain (pp) | $/run |
|---|---|---|---|---|---|---|---|
| qwen/qwen3.6-max-preview | 82.0 | 95.2 | 85.2 | 78.8 | −6.4 | 19.2 | 4.23 |
| google/gemini-2.0-flash-001 | 32.1 | 42.7 | 35.5 | 28.8 | −6.7 | 27.2 | 0.005 |
| inclusionai/ling-2.6-flash | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 |

ling-2.6-flash failed every cell with a 100% parse-error rate; every game ended as a first-move strict loss.

## Cost frontier

Pareto frontier of optimality vs. cost per run (no cheaper model scores higher): gemma-4-31b-it ($0.02, 43.7%), mercury-2 ($0.06, 64.8%), mimo-v2.5-pro ($0.45, 67.1%), kimi-k2.5 ($0.48, 71.7%), gpt-5.5 ($1.26, 82.1%). Best price-performance: kimi-k2.5 at $0.009 per optimal move; mercury-2 reaches ~65% at about $0.001 per optimal move. The cheapest models are heavily scaffold-dependent, so their cheap performance may not survive a different prompt.

## Prompt-family means (optimality %)

| Family | bench-a | benchx-a |
|---|---|---|
| strict-tag | 74.3 | 73.7 |
| no-examples | 69.2 | 69.2 |
| json | 62.3 | 61.5 |
| natural-language | 52.2 | 46.0 |
| minimal | 50.3 | 33.9 |

## Further results

- Error taxonomy: the compound IMPOSSIBLE class (wrong side, both sides, or an unreachable configuration) is 1.0% of turns under strict-tag in bench-a, 11.6% under minimal, and 21.7% under minimal in benchx-a. Parse errors cluster in natural-language (5.3% in bench-a). json contract compliance drops from 85% (bench-a) to 66% (benchx-a).
- Split-available states: strict-tag holds 96.7% legality (bench-a) and 96.1% (benchx-a); minimal manages 76.5% and 51.9%.
- Survival: under strict-tag in bench-a, 66% of games are fully optimal through the model's first move and 41% at the fifth; under minimal, 41% and 20%.
- Across 30 suites, strict-tag is the single best family 17 times; minimal is the single worst 16 times (17 with one tie).

## Method

Ruleset: chopsticks-mod5-dead-target-illegal-mirror-swap-illegal-repeat3-countdown100-v1. Two hands each with 0–4 fingers; 0 is dead. A turn is a tap (add a live hand to an opponent's live hand, mod 5) or a split (redistribute your own total; identity and mirror swaps are illegal). Both hands dead loses. A third repetition starts a 100-ply draw countdown. States are written from the mover's side as AB-CD; the model outputs the resulting state. No legal-move list is given.

Solver: retrograde analysis over all 1,250 ordered states, labelling each WIN, LOSE or DRAW for the mover, with optimal successors and distance to terminal.

Strict policy: an unparsable or illegal move ends the game as a model loss and is scored non-optimal.

Modes: bench-a (vs solver, stratified 59-state suite, context reset per game), benchx-a (vs solver, random starts, context accumulates), benchx-b (self-play over the stratified suite), bench-a-o (one ply from 11-11), benchx-o (one ply per suite state), bench-c (context-rot diagnostic); beta: bench-b, bench-lite; benchx is an alias for benchx-b. Wave 1 used bench-a and benchx-a.

Stratified suite benchx-stratified-v2: 59 positions; 28 WIN / 20 LOSE / 11 DRAW; distance 12 short / 16 medium / 20 long / 11 draw; legal-move count 23 low / 19 medium / 17 high; 42 with a dead hand / 17 without; 38 with a split available / 21 without.

Prompt families: strict-tag (full rules, 4 worked examples, <move> tags), no-examples (same without examples), json ({"move":"AB-CD","reasoning":"…"}), natural-language (loose instruction), minimal (one sentence of rules plus the bare state).

Metrics: optimal_rate_all (headline), legal_move_rate (gatekeeper), optimal_rate_legal, structure_gain (structured families minus low-scaffold families), example_gain (strict-tag minus no-examples), family range, model_score, $/optimal move. Headline rates carry Wilson score intervals.

## Limitations

One small, deterministic, fully solved game; no claims about general intelligence. Wave 1 used one 20-game run per cell (not the prescribed 59 games × 5 seeds), so small gaps such as the 80–82% top tier should not be over-read. The bench-a → benchx-a contrast confounds start distribution with history accumulation. benchx-o and bench-c were not collected in Wave 1. About 70% of costs are estimates (billed metadata for 45 of 150 runs). All traffic went through OpenRouter.

## Cite

The paper is in peer review. Until it is out, cite the project:

```bibtex
@misc{gendell2026chopbench,
  title  = {Chop-Bench: An Optimality-First Benchmark for Formal-Game Competence in Language Models},
  author = {Gendell, Guy},
  year   = {2026},
  month  = jul,
  note   = {Chop-Bench Project. Wave 1 (R1) interactive prompt-family sweep. Under peer review}
}
```