All posts
2 min readby Romiel Inolino

Ai2's XYBench: five LLMs focused on the user's underlying goal in only 35% to 41% of Stack Overflow answers

AI evalsLLM benchmarkssupport automationAI assistants

A new benchmark measures something support bots get wrong all the time: answering the question asked instead of the problem behind it.

What happened

Ai2, with researchers at Carnegie Mellon, NVIDIA and Johns Hopkins, released XYBench, a benchmark of 8,115 questions built around the XY problem. A person wants X, assumes approach Y will work, and asks about Y. The questions come from Stack Overflow (1,557), Stack Exchange (184), 102 hand-curated examples and 6,272 generated from WikiHow guides. The paper won the Best Resource Paper Award at EMNLP 2026.

The team tested Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro, one response per question. On Stack Overflow, answers addressed the stated request 75% to 86% of the time and the underlying goal 59% to 71% of the time. Only 35% to 41% of answers focused mainly on the underlying goal.

Two follow-up tests stand out. Giving Sonnet 4.6 the user's real intent raised its goal-solution rate from 59% to 67%, but its rate of identifying the misconception fell from 47% to 34%. And when the models were shown two answers, one literal and one addressing the real goal, they picked the goal-focused answer 71% to 84% of the time. They can recognize the better answer more often than they write it.

Claude Sonnet 4.5 served as the automated judge, and two human reviewers agreed with it 74.2% to 100% of the time depending on the metric. The benchmark covers English, single-turn, mostly procedural questions, and it doesn't include the newest frontier models.

My take

Intake bots and internal help assistants get XY questions constantly. Someone asks how to make a workflow run twice when the real issue is a webhook that fires late. A literal answer works and leaves the actual problem in place.

The gap between judging and writing suggests a practical fix: generate the answer, then run a second pass that asks whether it addresses the likely goal and rewrites it if not. I'd also build an eval set from your own past tickets where the first answer was technically right and still unhelpful. Whether newer models close this gap is something the paper can't tell you, so test on your traffic.

More posts