APPS
Evaluate Python programs generated from coding problems written in natural language.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
10,000 problems split evenly between training and testing, with introductory, interview and competition difficulty levels. Dan Hendrycks and Steven Basart are co-first authors. Full author team: Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song and Jacob Steinhardt.
Methodology
Run generated programs against problem-specific test cases. Test-case average gives partial credit; strict accuracy requires every test case to pass. Sampling count and difficulty subset must be reported separately. A small number of test cases can accept incorrect solutions. Paper revisions are not benchmark editions. Training data and held-out evaluation problems are separate.