Benchmark
Arena-Hard-Auto
Compare instruction-following models on difficult user questions and a separate creative-writing track.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
Arena-Hard-v2.0-Preview contains 500 hard questions and 250 creative-writing questions. It follows the original 2024 Arena-Hard-Auto benchmark; paper revision v2 is not benchmark version 2.0.
Methodology
Automatic judges approximate human preferences. The documented hard-question configuration uses style control and Gemini 2.5, with GPT-4.1 as an alternative. Creative writing uses an ensemble of both judges. Keep the track, judge and configuration aligned when comparing results.