Stubbi
Rankings
Benchmark

Airbnb Production Classifier Evaluation

Compare classification quality, latency and cost for a fine-tuned model and general-purpose models on one production policy task.

Checked Nov 19, 2026Airbnb Engineering study

Explore the original evidence

Explore the original evaluation, its methods, and reported findings.

Open eval

Coverage

A pricing-and-payment-policy classifier evaluated on 10,000 held-out, weighted examples. This is a published production case study using private evaluation data, not a downloadable benchmark suite.

Methodology

Compares max F1 and server-side latency under concise and detailed instructions. Probability-producing models use optimized thresholds; verdict-only models supply a fixed classification point. Cost estimates use October 2026 list prices, a stated cache assumption and a single-GPU hosting baseline.

People & organizations

1

Sources

HN discovery story