Which AI model tags SEC filings best?

Each model got Panabee's production tagging prompt as the system message and one filing as the user message. The filing text is exactly what the production tagger sends: a short context block with the date, ticker and company, then each document converted from HTML to Markdown. The model answered all 41 tags yes or no in one JSON object, with an evidence quote and a reason for each yes. Only yes answers count.

The 100 filings are from May to November 2026. 50 are quarterly earnings: the earnings press release plus the full 10-Q, with a median of about 42,600 tokens. Half of those were drawn at random and half were picked for rare situations and hard negatives, such as reaffirmed guidance. The other 50 are event 8-Ks with their press releases, with a median of about 4,600 tokens, including routine filings that should get no special-situation tags.

Language models ran through OpenRouter at temperature 0 with medium reasoning. Decision models return a probability for each yes-or-no question. They got the same text and one question per tag, and a probability of 0.5 or more counts as yes. Neither decision model returns an evidence quote.

Scores pool every filing and tag in a group, so each true tag counts once. The production converter wraps table cells in background-color markers that carry no content, so they were removed. Rerunning two models with the markers left in changed no score beyond run-to-run noise. Running the whole benchmark cost $14.86 in model calls.

Two models labeled every filing with the same prompt: Gemini 3.8 Flash and GPT-6.1 Terra. Claude Code then adjudicated 339 calls by reading each filing against the written tag definitions. It ruled on every tag either labeler said yes to, every disagreement, and every yes from a tested model that both labelers said no to. Each ruling quotes the filing, and a script confirms the quote appears in it word for word. A tag nobody flagged counts as no.

The rulings came out 146 yes, 132 no and 61 ambiguous. A separate Claude review of 52 sampled rulings found none to overturn. No person reviewed the labels.

Three judgment calls shape the labels. A company that doubts it can continue as a going concern but has no debt is left out of the financial distress score. Medical device makers are not tagged biotech. Guidance narrowed to a range with a lower midpoint, as in one EDAP earnings release, counts as lowered guidance.

• Gemini 3.8 Flash and GPT-6.1 Terra made the labels and Claude adjudicated them, which could favor those model families. A tag both labelers missed was caught only if a tested model flagged it.
• Coverage is thinner than planned, with 62 true priority tags. Raised guidance, going-concern warnings, financial distress, pending acquisitions, lowered guidance and record revenue carry most of the priority score.
• Earnings filings stop at 55,000 tokens, which leans toward smaller companies. The long-filing check is the only test above that.
• These are November 2026 model versions with one prompt. A different prompt could change the order.

Not run: Muse Spark 1.3 Contributor. Its only provider may train on prompts, and our gateway account blocks providers that do. GLiDE. No gateway we use carries it. It needs a separate account, and it bills each filing once per tag.

The download holds 80 filings with their labels and evidence, the prompt, the scoring code and every model's raw answers. Another 20 filings are held out so future models can be tested on filings they have not seen. Their scores count in the tables above, but their text is not published.

Download the data 2.3 MB zip

SHA-256 6fe6d397e2005ed8e6fbf6c6107d97d99508ac5367e84b2cd7bd247653047250