Benchmark
Octobench
Evaluate coding agents on real repository changes with held-out tests.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
The current repository describes 80 one-shot tasks and 25 long-run sequences across five programming languages. The earlier report covers 25 tasks; these are different snapshots of the same project.
Methodology
Agents modify pre-fix repositories. Held-out project tests determine completion, while a separate model judge assesses work quality. Cases must demonstrate failure before the reference fix and success afterward. Muvon also develops the evaluated Octomind agent; results are publisher-reported.