Zapier tested AI agents on 657 business workflows. The best model passed 51.3%, and most failures looked like success

The most useful AI benchmark for automation builders this year checks the CRM, not the chat reply.

What happened

Zapier added AutomationBench to Fireworks' Specialized Intelligence Index. The benchmark is built on patterns from more than 2 billion monthly tasks across 3.7 million companies on Zapier. Each task sets up a small simulated company with a CRM, an inbox, spreadsheets and support cases, spread across 47 simulated apps. The agent gets one request, like an ops hire would on Slack, and cannot ask clarifying questions.

One example: mark a Meridian Corp deal as won and notify the right team. The CRM has three accounts with nearly identical names, the tier sheet has an old row and a newer one, and the routing rule sits in an email. Zapier checks what changed in each app, not what the agent says it did. The leaderboard runs on 657 private tasks.

Results: Gemini 4 Argon leads at 51.3%, Claude Sonnet 5.5 is second at 44.7%. Zapier says the most common failure is false confidence, where the agent reports success while the data says otherwise. Price did not predict performance: GPT-6 Sol scored about 80% of GPT-6 Astra for roughly a sixth of the cost per task, and Sonnet 5.5 beat Opus 5.5 while costing less. GPT-6 Luna and DeepSeek V4 Flash cost $0.04 a task but completed fewer than a quarter of full workflows.

My take

This matches what I see in client builds. The dangerous failure is not an error. It is a run that finishes green and updated the wrong record.

So I treat the AI step as a proposal and the next step as the audit:

• Use cheap models for single steps like tagging a ticket or cleaning text. Pay for a stronger model only where the agent has to find records across apps.
• After any AI step that touches customers or money, search the record again and filter on the expected values. Stop the Zap or n8n run if they do not match.
• Make lookups unambiguous. Pass record IDs between steps instead of names, and keep routing rules in a table, not in an email thread.
• Before swapping in a new model, replay a handful of real past runs and compare the end state.

Half of the best model's full workflows still fail. Design for that and the other half becomes very useful.