Ai2's StateBench: frontier models find the latest value fine, but drop to 13% or lower when updates build on each other in long inputs

If your automation asks a model to work out the current state from a long history, this benchmark is worth five minutes.

What happened

Ai2 released StateBench, a benchmark for how well language models keep track of values that change across long documents. It sorts problems into four levels, explained in programming terms. R-trivial: a value is set once and never changes. Aperiodic: a value is reassigned, so only the latest matters. Soluble: each update builds on the current value, like x = x + 5 or a running game score. General finite: one variable takes another's value, like swapping two variables.

Ai2 tested four frontier models: Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna. All four did well on the first two levels. On soluble and general finite problems, accuracy fell sharply as inputs grew longer, dropping to 13% or lower on the longest general finite problems. The team suggests one factor is that many more steps can affect the final answer, so one missed update breaks the result.

Format mattered too. Some small models scored near the ceiling on a standard retrieval style version of a problem but near chance when the same problem was framed as spreadsheet edits.

My take

Map those levels onto real client work. "What is the current deal stage?" is a latest value question, and models handle it. "What is this customer's balance after 40 credits and refunds?" is a running total. "Who owns this lead after three reassignments and a territory swap?" is assignment by reference. Those are the harder levels, and they are common in CRM and finance automations.

The fix is not a bigger prompt. It is moving state out of the model:

1. Compute running totals, balances and ownership in code or in CRM fields.
2. Pass the model the current state, not the full change log.
3. If the model must read a history, ask for the list of changes it found, then do the math in a code step and compare.

Also test with your real data format. The benchmark shows the same logic can pass in one format and fail in another, so a demo on clean text proves little about a messy export.