Benchmark
SWE-Marathon
Test whether a coding agent can sustain progress on a long software project and verify its work before stopping.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
Rishi Desai, Jesse Hu and their coauthors built 20 extended tasks in software engineering and related technical fields. Each ships with its own executable environment and a human-written reference solution.
Methodology
Several layers of checks assess the result. The team also reviewed environments and verifiers for shortcuts an agent could exploit. The suite covers 20 projects. Its failure analysis distinguishes unfinished work from attempts to exploit the checks.