Stubbi
Rankings
Benchmark

Arena-Hard-Auto

Compare instruction-following models on difficult user questions and a separate creative-writing track.

Checked Nov 19, 2026Original benchmark repository

Explore the original evidence

Explore the original evaluation, its methods, and reported findings.

Open eval

Coverage

Arena-Hard-v2.0-Preview contains 500 hard questions and 250 creative-writing questions. It follows the original 2024 Arena-Hard-Auto benchmark; paper revision v2 is not benchmark version 2.0.

Methodology

Automatic judges approximate human preferences. The documented hard-question configuration uses style control and Gemini 2.5, with GPT-4.1 as an alternative. Creative writing uses an ensemble of both judges. Keep the track, judge and configuration aligned when comparing results.

People & organizations

1

Sources

Original paper and eight-author attributionReviewed aggregator alias