Benchmark
SWE-Lancer
Evaluate freelance software fixes and choices between implementation proposals.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
Paid freelance software tasks from the original study; the July 17, 2025 runnable release retains 198 offline-compatible tasks from an earlier 237-task subset.
Methodology
Coding tasks use engineer-written end-to-end tests. Manager tasks check the selected proposal against the recorded choice; historical task payouts support a separate dollars-earned metric.