Benchmark
CyberGym-E2E
Test agents discovering, reproducing and patching vulnerabilities in real software projects.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
920 vulnerabilities across 139 open-source projects. The project distinguishes an earlier 615-task evaluation set and a separate patch-only setting.
Methodology
An agent investigates a codebase, produces a crash-inducing input and writes a patch. Success requires reproducing the crash, fixing it and preserving functionality tests. A fourth diagnostic checks whether the patch fixes the specific ground-truth vulnerability. Artificial Analysis evaluates a separate 131-task selection with Stirrup and a 90-minute limit; keep its results separate.