# Chop-Bench research notes

*Short version · Wave 1 (R1 sweep, archive `r1-4-26-26`) · last edited 23 October 2026*

The paper, *Chop-Bench: An Optimality-First Benchmark for Formal-Game Competence in Language Models*, is in peer review. Until it's through, these notes are a short summary; the full methods and results will follow the paper. Wave 2 is coming soon.

To edit: change `public/research.md` and deploy.

## 1. The idea

Chop-Bench grades every move a language model makes in a fully solved variant of Chopsticks against an exact solver. A move is either legal or not, and either solver-optimal or not. Winning matters less than getting each move right.

## 2. The game

- Two hands each, 0–4 fingers; 0 is dead.
- **Tap:** add one of your live hands to one of the opponent's live hands, mod 5.
- **Split:** redistribute your own total. The identity split and the mirror swap are illegal.
- Both hands dead loses. A third repetition starts a 100-ply draw countdown.

States are written from the mover's side as `AB-CD`. The model outputs the resulting state, with no list of legal moves given. There are 1,250 ordered states, 98 of them terminal.

## 3. The solver

The solver uses retrograde analysis, the backward induction used to build endgame databases:

1. Start from the terminal positions and work backwards through the move graph.
2. A position with a move into a lost position for the opponent is a **WIN**.
3. A position where every move leads to a won position for the opponent is a **LOSE**.
4. Whatever the propagation never reaches is a **DRAW**: neither side can force an end.

It also records the plies to the end under best play. A move is scored optimal if it:

- wins as fast as possible, in a WIN position;
- keeps the draw, in a DRAW position;
- loses as slowly as possible, in a LOSE position.

The full verdict table, all 625 positions from the mover's side as `[outcome, distance, optimal moves]`, is at [`/data/solver.json`](data/solver.json). The "Take the benchmark" game on this site grades you with that exact table.

## 4. Grading a move

1. **Parse.** Extract a state token from the reply.
2. **Check legality.** An illegal move gets an error class, such as a tap onto a dead hand or a split that changes the total.
3. **Check optimality.** Compare the move with the solver's optimal set.

Under the strict policy, an illegal or unparsable move ends the game as a loss and counts as non-optimal. That makes legality a gatekeeper.

## 5. The leaderboard numbers

A model's headline optimality is averaged over the prompt families in each mode, then over the two modes. A model scores well only by doing well everywhere.

The ± in nerd mode is a 95% interval from turn-level sampling. Each cell's binomial variance is carried through those two averages, with the Agresti–Coull adjustment so cells at 0% or 100% still count.

Three ways to read the intervals:

- **They're optimistic.** Moves within a game are correlated, so treat each ± as a lower bound on the real uncertainty.
- **The top three are tied.** GPT-5.5 (82.1 ± 2.5), Qwen3.6-Plus (80.0 ± 2.7) and Gemini 3.1 Pro (79.2 ± 3.1) can't be separated in Wave 1.
- **Prompt sensitivity is separate.** Spread across prompt families is reported as structure gain and family range. It describes the model, so it isn't treated as noise.

`tools/intervals.py` rebuilds the intervals from the run archive.

## 6. Limits

- **One game.** Wave 1 covers a single small, fully solved game, so there are no claims about general intelligence.
- **One run per cell.** Each cell was one 20-game run, so small gaps shouldn't be over-read.
- **One route.** All traffic went through OpenRouter.

## Changelog

- **23 Oct 2026.** Cut to a short summary while the paper is in peer review.
- **23 Oct 2026.** First version: 95% intervals and scored-turn counts from the archive. The site's game now grades against the exported solver table.