Stubbi
Rankings
Benchmark

APPS

Evaluate Python programs generated from coding problems written in natural language.

Checked Nov 17, 2026Original APPS project

Explore the original evidence

Explore the original evaluation, its methods, and reported findings.

Open eval

Coverage

10,000 problems split evenly between training and testing, with introductory, interview and competition difficulty levels. Dan Hendrycks and Steven Basart are co-first authors. Full author team: Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song and Jacob Steinhardt.

Methodology

Run generated programs against problem-specific test cases. Test-case average gives partial credit; strict accuracy requires every test case to pass. Sampling count and difficulty subset must be reported separately. A small number of test cases can accept incorrect solutions. Paper revisions are not benchmark editions. Training data and held-out evaluation problems are separate.

People & organizations

2

Sources

Original paper and revision history