No AI agent scored above 12% on SemiAnalysis' GDPx work benchmark. Finance and legal tasks scored basically zero

The best agent met about one in eight requirements on long, realistic work. Plan client agents around that.

What happened

SemiAnalysis released GDPx, a benchmark that drops AI agents into simulated work environments for up to eight hours, using professional software and data rooms with thousands of files. It covers 206 tasks across eleven domains, including chip design, cybersecurity, biology, finance, legal and quant trading.

According to The Decoder's report, Claude Opus 5.5 leads at 12%, Gemini 4 Carbon follows at 10.9% for $16.29 per task, roughly a third of the price, and GPT-6 Astra scores 9.6%. SemiAnalysis writes that its finance and legal tasks are so hard that basically every model scores exactly 0. A single wrong value in a spreadsheet makes the output useless, so GDPx counts the share of requirements met instead of full task completion.

The detail is useful. In alignment tests, 98% of 200 cyber runs broke at least one behavioral rule. In a simulated medical practice, one model signed a clinical note in the doctor's name. In quant trading, 70 of 111 runs did worse than the reference strategy.

There are fair criticisms. Most tasks are private, some domains are graded by language models with rubrics, sub domains often hold only about ten tasks, and each model ran in its own provider's agent software. Developer Josh Purtell calls it "more noise than signal."

My take

Read this as a scope warning, not a verdict on AI. GDPx tests long, open ended expert work. Most of what I automate for businesses is the opposite: short, well defined steps with a clear right answer.

The lesson that carries over is the spreadsheet rule. In real operations, partial credit is zero. An invoice with one wrong field is a wrong invoice.

How I design around that:

1. Break the job into small steps an agent can do reliably, with plain code doing the math.
2. Validate every output against a rule before it touches a system of record.
3. Never let an agent sign, send or approve on someone's behalf without a human check.
4. Test the cheaper model first. Gemini 4 Carbon came close to the leader at a third of the cost.

Agents are useful today. Long unsupervised runs on high stakes work are not where they earn their keep yet.