Benchmark
Airbnb Production Classifier Evaluation
Compare classification quality, latency and cost for a fine-tuned model and general-purpose models on one production policy task.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
A pricing-and-payment-policy classifier evaluated on 10,000 held-out, weighted examples. This is a published production case study using private evaluation data, not a downloadable benchmark suite.
Methodology
Compares max F1 and server-side latency under concise and detailed instructions. Probability-producing models use optimized thresholds; verdict-only models supply a fixed classification point. Cost estimates use October 2026 list prices, a stated cache assumption and a single-GPU hosting baseline.