Benchmark
CloudOpsBench
Can an agent leave cloud infrastructure in the state an operator requested?
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
The project runs 45 infrastructure tasks four times for each combination of model, reasoning effort and harness. Antonio Mejia leads the benchmark with contributors from multiple organizations.
Methodology
Each trial starts with fresh resources. After the agent works, a checker inspects the final state. Timeouts count against the resolution rate. Infrastructure errors are excluded. The setup can use a real account or the Vera emulator.