Benchmark
BigCodeBench
Evaluate Python programs that combine libraries and follow complex instructions.
Published leaderboard
Explore scores and evaluation settings on the publisher’s leaderboard.
Open leaderboardCoverage
1,140 programming tasks in the Full set and a separate Hard subset. Complete and Instruct use different prompt formats.
Methodology
Execute generated programs against tests. The official board reports calibrated pass@1 under its stated decoding and evaluation configuration.