CloudOpsBench, a new open-source benchmark developed by experienced DevOps engineers led by Ang Chen at the University of Michigan, reveals that reasoning is the biggest bottleneck for artificial intelligence in cloud operations. Version 1.0 of the benchmark includes 45 tasks on AWS, such as fixing services, removing resources, restricting access rights, rolling out updates, and decommissioning projects.
Testing shows that even the best models fail at many of these tasks, with Claude Fable 5.1 solving 60 percent and GPT-6 Astra managing 53 percent. The research team found that roughly 80 percent of failures occur because AI agents misjudge the scope of a task, fail to check relevant services, or stop too early, while about 20 percent of errors involve wrong actions with potentially irreversible infrastructure consequences.
All benchmark tasks run identically in the real AWS cloud and in an emulator, with planned extensions for GCP, Azure, and OCI. Released under an Apache 2.0 license on GitHub alongside a detailed failure analysis, this project helps measure and address current limitations in AI agent reliability.