Benchmark
HumanEval
Evaluate whether generated Python functions pass their tests.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
Handwritten Python programming problems with function signatures, descriptions, and unit tests.
Methodology
Execute generated solutions against tests and report pass@k for a stated number of samples and generation settings.