Having AI advice available nearly wiped out people's willingness to say I don't know, even when the AI was wrong

Human in the loop only works if the human still feels allowed to say "not sure".

What happened

A preprint covered by The Decoder ran five experiments with 3,132 participants. The questions were chosen so the AI advice was almost always wrong, which separates the effect of AI access from AI accuracy.

Without AI, participants declined to answer 36 and 44 percent of questions in the first two studies. With AI access, that fell to 6 and 3 percent. In another study, confidence rose from 29.6 to 75.9 out of 100 while correct answers fell from 27.6 to 10.0 percent. Across studies without incentives, people with AI got 9.2 percent right versus 27.5 percent without it.

Paying for correct answers and penalizing wrong ones helped somewhat: people asked the AI less and were more accurate, but still abstained far less than the no AI group. Showing the AI answer automatically, without anyone asking for it, barely changed the effect. The authors note the questions were movie trivia, and whether the effect is as strong elsewhere is still open.

My take

A lot of automations I see use AI to pre-fill something a person then approves: a lead score, a reply draft, a refund decision, a categorized ticket. We call that human in the loop and assume it catches errors. This study suggests the pre-filled answer can quietly remove the reviewer's instinct to say "I don't know".

How I would design around it:

1. Make "unsure" a real button with a real route, like escalate or request more info, not just approve or reject.
2. Show the AI's evidence next to its suggestion so reviewers judge the source, not the confidence of the wording.
3. For high stakes fields, let the person answer before revealing the AI suggestion.
4. Track how often reviewers override the AI. An override rate near zero is a warning sign, not a success metric.

Which approval step in your business gets rubber stamped the most? Send me a message about it through romielwillautomate.dev and I will tell you how I would rebuild that review so the human actually reviews.