Benchmark
aws-bench
Evaluate agents diagnosing and operating real AWS infrastructure.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
Versioned datasets including a nine-task quickstart, 78 basic tasks and 47 advanced tasks.
Methodology
Provision isolated scenarios. Use a model judge for read-only diagnosis and programmatic checks of live infrastructure for resource changes.