The best AI agent finished 29 of 45 real AWS tasks. About 80% of failures came from stopping too early, not from missing knowledge

Every command succeeded, the agent's own re-check came back clean, and the task was still not done.

What happened

CloudOpsBench is a new open source benchmark from a team led by Ang Chen at the University of Michigan. Version 1.0 has 45 DevOps tasks on AWS: decommissioning projects, securing access, migrating services, configuring monitoring and repairing failures. It runs the same way in real AWS and in an emulator.

The strongest agent, Claude Fable 5.1, completed 29 of the 45 tasks. GPT-6 Astra completed 26.

The failure analysis is the useful part. No task failed because an agent misunderstood a cloud service. About 80% of failures were reasoning gaps: agents stopped surveying before they had checked every service a DevOps engineer would check, and declared the task complete. In one task, deleting every queue in a project, every model knew that MediaConvert can have its own queues, yet not one attempt listed them.

The other 20% were wrong actions. In a bucket migration task, agents found an in progress multipart upload, treated it as leftover data and aborted it. That data cannot be recovered.

My take

The same pattern shows up in business automation. Agents rarely fail because they do not know HubSpot or Airtable. They fail because they decide the job is smaller than it is, and nothing in their output tells you something was missed.

So "done" should never be the agent's call alone:

1. Give the agent the full scope as a checklist: every object, list, pipeline or table in play. Do not make it discover the scope.
2. Verify with a separate step that queries the system fresh and compares counts, instead of trusting the agent's summary.
3. Put any delete, archive or abort behind an approval or a dry run first.
4. Log what the agent looked at, not just what it changed, so gaps show up in review.

Point 1 alone would have caught the MediaConvert queues. Cleanup jobs in CRMs work the same way: the duplicates you did not list are the ones that stay.

If you have an agent or cleanup job that keeps saying "done" when it is not, message me with what it is supposed to do. I will tell you where I would add the checks. romielwillautomate.dev