Benchmark
Big Bench Audio v1.2
Test reasoning through spoken questions and spoken answers.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
1,000 questions adapted from four BIG-Bench Hard categories using 23 synthetic voices. The original audio benchmark dates to December 2024; v1.2 dates to May 2026. Exact release days are not specified.
Methodology
Native audio models receive spoken questions and return spoken answers, which a model judge grades for correctness. Version 1.2 updates the judge and grading harness to better recognize correct answers in verbose responses.