Anthropic gave 50 experts an agent with coin toss opinions. They followed it up to 86% of the time and felt the calls were their own

Your human review step may not be doing what you think it does.

What happened

Anthropic had 50 experts in causal inference analyze data from a fictional farming cooperative and recommend whether to expand an advisory program. At each of five decision points there were two valid options, and a coin toss set which one the AI agent preferred. The agent could not lie or hide data. It only had to make the strongest honest case for its side.

According to The Decoder's report on the study, participants followed the agent's preference in up to 86% of the four analytical decisions. Having an opinion first offered little protection: an opposing agent view made people 53 percentage points more likely to switch. Experience, expertise and trust in AI showed no reliable link to the effect. Afterward, 52% said the final recommendation felt more like their own, and another 35% said it was mostly or entirely theirs. The agent stated its position first in 77% of decisions, and usually argued only its own side.

The upside was real too. With AI, 3 of 50 analyses had serious flaws, against 6 of 11 without it, and median time fell from 90 to 65 minutes. The researchers suggest agents argue both sides or ask for the person's view first.

My take

Swap "experts" for "sales reps" or "ops managers" and this is every CRM workflow where AI suggests a lead score, a next step or a deal priority. The human clicks approve, and we call it human in the loop.

What I would change in those builds:

• Capture the human's call first. Ask for their pick before showing the AI's.
• Make the AI show the other side. One line on why a reasonable person would choose differently.
• Track override rates. If people change almost nothing, you have a rubber stamp, not a review.
• Decide which calls stay human. Pricing exceptions, refunds and anything legal should not drift to the model by default.

Keep the speed. Just make sure the judgment is still yours.