Which AI model analyzes earnings filings best?
8 AI models each read the earnings release and 10-Q for 15 real companies and wrote the analyst reports a financial news service publishes. Graders checked every figure against the filings, blind to which model wrote what.
We ran this benchmark to choose the model that writes the earnings analysis for Panabee.
Claude Haiku 5.5, out of 10, from the grader's own model family
GPT-6.1 Terra, per event, against 1.5 for Gemini 3.5 Flash-Lite
GPT-6 Luna a month, against $159 for Gemini 3.5 Flash-Lite
Ranked results
Higher is betterOverall is the grader's judgment of how useful and trustworthy the reports are to an investor, from 1 to 10, averaged over the 8-K and 10-Q reports and the events. Accuracy weighs most: a sharp report with a wrong figure can score below a plainer correct one. Ranked first counts the events where a model came top. GPT-6.1 Terra made the fewest critical errors.
| Rank | Model | Overall | Ranked first | Critical errors per event | Per analyst call | Per month | Seconds per call |
|---|---|---|---|---|---|---|---|
| 1 | Claude Haiku 5.5 | 7.9 | 11 of 15 | 0.3 | $0.0753 | $395 | 188 |
| 2 | GPT-6.1 Terra | 7.5 | 1 of 15 | 0.0 | $0.0965 | $506 | 52 |
| 3 | GPT-6 Luna | 6.8 | 2 of 15 | 0.3 | $0.0137 | $72 | 119 |
| 4 | DeepSeek V4 Pro | 6.3 | 0 of 15 | 0.3 | $0.0612 | $321 | 226 |
| 5 | DeepSeek V4.1 Flash | 6.3 | 1 of 15 | 1.7 | $0.0112 | $59 | 138 |
| 6 | Qwen 3.8 Flash | 5.4 | 0 of 15 | 1.8 | $0.0171 | $90 | 152 |
| 7 | Gemini 3.8 Flash | 4.4 | 0 of 15 | 2.6 | $0.0914 | $479 | 81 |
| 8 | Gemini 3.5 Flash-Lite | 3.6 | 0 of 15 | 1.5 | $0.0304 | $159 | 13 |
Per analyst call is the measured cost of one report, including thinking tokens. Per month is the cost of Panabee's earnings analyst at its volume, about 5,200 reports a month. Seconds per call is the median, through OpenRouter on each model's listed host. Gemini ran in JSON mode as in production. The other models returned plain text that production's parser reads.
Scores by skill
1 to 5, higher is betterAccuracy counts errors found against the filings. Lead asks whether the first facts are the ones that move the company and its stock. Guidance asks whether every guidance change is exact and its implications computed. Insight asks whether the report makes the points a specialist would. Concision penalizes padding and repetition.
| Rank | Model | Accuracy | Lead | Guidance | Insight | Concision |
|---|---|---|---|---|---|---|
| 1 | Claude Haiku 5.5 | 4.1 | 4.4 | 4.5 | 4.8 | 3.1 |
| 2 | GPT-6.1 Terra | 4.8 | 4.5 | 4.3 | 3.9 | 4.5 |
| 3 | GPT-6 Luna | 4.2 | 4.4 | 3.9 | 3.6 | 3.6 |
| 4 | DeepSeek V4 Pro | 4.0 | 4.2 | 3.7 | 3.3 | 3.3 |
| 5 | DeepSeek V4.1 Flash | 3.2 | 4.5 | 4.1 | 4.3 | 2.9 |
| 6 | Qwen 3.8 Flash | 3.2 | 4.3 | 3.9 | 3.5 | 2.7 |
| 7 | Gemini 3.8 Flash | 2.8 | 4.0 | 3.5 | 2.8 | 3.0 |
| 8 | Gemini 3.5 Flash-Lite | 3.3 | 3.2 | 2.3 | 2.0 | 3.1 |
What the difference looks like
Quoted from the reportsTenax Therapeutics
The drug missed its main goal. The 34-meter benefit came from a group of patients picked after the results were in.
- Gemini 3.5 Flash-Lite
In a prespecified subgroup of more symptomatic patients making up over 50% of the trial, treatment yielded a statistically significant 34-meter improvement in 6-minute walk distance.
- DeepSeek V4.1 Flash
the 34-meter effect is derived from arm-specific regression models and classification trees fitted to the same LEVEL dataset used to select the criteria. Reported p-values are nominal and the confidence interval extends to 15 meters, so the estimate likely overstates the benefit a fresh population will show
- GPT-6.1 Terra
The 34-meter estimate comes from criteria selected after reviewing LEVEL data. Its nominal statistical result does not account for the risk of finding an apparent benefit through exploratory analysis.
Gemini 3.5 Flash-Lite turns a result found after the fact into a planned, significant one. The other two explain why the next trial will likely show a smaller effect.
Root
Root's combined ratio improved 4.9 points, mostly because a one-time warrant charge a year earlier did not recur.
- Claude Haiku 5.5
The 4.9-point net combined ratio improvement is mostly lower Carvana warrant expense, which accounts for roughly $15 million of the $20 million net income swing, while Adjusted EBITDA was flat Y/Y at $34 million.
- DeepSeek V4 Pro
Profitability inflection: net income swung to $14 million Y/Y from a $5 million loss, while the net combined ratio improved 4.9 points to 97.2% and annualized ROE reached about 18%.
- Gemini 3.5 Flash-Lite
Combined ratio enhancement: Net combined ratio improved 4.9 percentage points year-over-year to 97.2%, driven by disciplined expense management and lower operating overhead.
Only Claude Haiku 5.5 finds that most of the gain is a one-time item. The others credit the business.
Stewart Information Services
The release called the title loss ratio comparable, but it rose to 3.4% from 3.0%.
- Qwen 3.8 Flash
Title loss ratios held steady even as loss dollars grew 20%, showing underwriting control.
- GPT-6.1 Terra
Loss expense rose to 3.4% of title operating revenue from 3.0% a year earlier. At the prior-year rate, the quarter’s provision would have been approximately $3 million lower
- Claude Haiku 5.5
The higher loss ratio accounts for about $3 million of the $9 million adjusted decline, although management describes the ratio as comparable.
Qwen 3.8 Flash repeats the release and calls it underwriting control. The other two measure what the higher loss ratio cost.
Speed depends on the host
DeepSeek V4.1 FlashOpen models run on many hosts at very different speeds. The same DeepSeek V4.1 Flash calls, two events with both reports, ran on five hosts that do not train on prompts.
| Host | Tokens per second | Seconds per call | Per month |
|---|---|---|---|
| DeepInfra, fp8 (used in the tables) | 120 to 160 | 115 to 275 | $60 |
| DeepInfra, priority tier | 120 to 130 | 206 to 252 | $100 |
| DigitalOcean | 300 to 440 | 34 to 97 | $120 |
| Together | 280 to 370 | 47 to 76 | $200 |
| Baseten, fast tier | 880 to 920 | 24 to 29 | $430 |
DeepSeek writes several times more thinking tokens than Gemini Flash-Lite at the same setting, so even its fastest host takes longer per call. Monthly costs scale each host's measured cost on these calls. DigitalOcean, Together and Baseten do not publish their precision, and their quality was not graded.
How the test works
Each model ran Panabee's two production earnings analysts on 15 events from November 3 to 6, 2026. The first reads the 8-K earnings release and its exhibits, with Panabee's report on the prior quarter as context. The second reads the full 10-Q, with the prior quarter's report and the 8-K report the same model just wrote.
Panabee's own code built every request, so the documents, their conversion from HTML to Markdown and the context blocks match production exactly. Only the model changed. Every model ran through OpenRouter on its maker's own endpoint or one named host, with no fallback, a filter against hosts that train on prompts, and high reasoning effort.
The events were drawn one per situation or sector: lowered, missed and beaten guidance, a going-concern warning, a strategic review, a margin inflection, a bank, a REIT, an insurer, a biotech, software, industrials, energy and consumer companies.
Running the models for this page cost $11.94.
How the grading works
Claude Code subagents graded each event in two packets, one with every model's 8-K report and one with every model's 10-Q report, under letters shuffled per packet. Graders never saw which model wrote what.
Graders checked every figure in each report's key facts, every guidance statement and every derived figure, such as growth rates or the quarter implied by full-year guidance, against the filings, plus spot checks. Every error is quoted with the filing's evidence. A critical error is a wrong or invented figure or claim a reader could act on, or a result called significant that was not. A minor error changes no conclusion.
A model's score on an event averages its two packets. Its rank on the event is the mean of its two packet ranks.
Limits
- One Claude grader per packet, with no human review. Claude Haiku 5.5 is in the same model family as the grader.
- 15 events from one week of third-quarter results, mostly small and mid-sized companies. Scores between models a few tenths apart are within the noise. In an earlier round, regrading 5 of these events with fresh graders swapped two close models once.
- One prompt, Panabee's, written for Gemini. A prompt tuned for another model could change the order.
- Every model saw Panabee's report on the prior quarter, which carries its own errors.
- Speed was measured on a weekend and may differ under earnings-season load.
Models and providers
| Model | Version | Served by | Thinking tokens per call |
|---|---|---|---|
| Claude Haiku 5.5 | anthropic/claude-haiku-5.5 | Anthropic | 40,504 |
| GPT-6.1 Terra | openai/gpt-6.1-terra | OpenAI | 3,019 |
| GPT-6 Luna | openai/gpt-6-luna | OpenAI | 10,645 |
| DeepSeek V4 Pro | deepseek/deepseek-v4-pro-0813 | Baidu | 12,087 |
| DeepSeek V4.1 Flash | deepseek/deepseek-v4.1-flash-1017 | DeepInfra | 18,613 |
| Qwen 3.8 Flash | qwen/qwen3.8-flash | Alibaba | 15,580 |
| Gemini 3.8 Flash | google/gemini-3.8-flash | Google AI Studio | 9,722 |
| Gemini 3.5 Flash-Lite | google/gemini-3.5-flash-lite | Google AI Studio | 3,389 |
The dataset
The download holds 12 events with the filing text every model read, the prior-quarter report, every model's reports, and the grades with every error quoted. Another 3 events are held out so future models can be tested on events they have not seen. Their scores count in the tables above, but their text is not published.
SHA-256 75dd36c5188b7789c8eef81933e8baaab4f594468c9be832e6b0c00bf169d2d4