Benchmark
APEX-SWE
Test agents on building integrated software systems and diagnosing production-style failures.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
Mercor created 200 held-out tasks, split evenly between integration and observability. Cognition reviewed a subset to help reflect professional engineering work.
Methodology
Human-written rubrics and unit tests assess function, robustness and code style. The board reports first-attempt success and identifies the evaluation harness. The 50 public development cases are separate from the hidden leaderboard set. Preserve harness settings when comparing results.