The best coding agent met about 70% of requirements on real user change requests, but fully finished only 26.4% of tasks

Coding agents now do most of the job. The problem is the part they leave out.

What happened

Researchers from Alibaba and several Asian universities released Co-Design Bench, a benchmark of 273 tasks curated from more than 26,000 real user coding sessions. Agents must revise existing web apps from real change requests, sometimes using images, audio, video or documents as input, like pulling an amount from a receipt and updating every dependent total. The Decoder reports the team favored requests where users had to rephrase or roll back changes.

Grading is strict: one AI judge tests the finished app in a browser, and a separate verifier challenges every positive verdict and looks for hardcoded results. Per the paper, the strongest of ten models passes an average of 69.9% of criteria per task but fully satisfies only 26.4% of tasks. The Decoder names GPT-6 Astra as the leader, ahead of Claude Opus 5 and Qwen3.8-Max. Every model scored lower when a task combined three or four types of change.

Two more findings stand out. GLM-5.2 used nearly six times as many tokens as GPT-6 Astra on smaller changes and still scored significantly worse. And the harness mattered: Qwen3.8-27B scored 75% or 56% on larger overhauls depending on the environment. The authors recommend reporting model and harness together.

My take

"Mostly done" is the most expensive state in automation work. A workflow that updates nine of ten fields correctly looks finished until the tenth field breaks an invoice.

What I take from this for client builds:

1. Write acceptance criteria before the agent starts, one line per requirement, and check each one. That is exactly how this benchmark exposed the gap.
2. Split requests that mix layout, logic and content into separate tasks. Performance dropped as change types piled up.
3. Evaluate the model and the harness as a pair. Swapping one without retesting is a new system.
4. Don't buy your way out with more tokens. Here, more compute did not fix the weakness.

The authors note limits: one fixed request per task, no mid task corrections, and AI judges that can err. Still, plan for a human cleanup pass.