Benchmark
Vespper DOCX Editing Evaluation
Compare document-editing tools used by an AI agent on legal work from Harvey LAB.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
A stratified sample of 180 document-editing tasks from 360 matching Harvey LAB tasks compares SuperDoc MCP 1.5.0, Office CLI 1.0.155, Adeu MCP 3.0.7, Stemma MCP 0.6.0 and a python-docx baseline.
Methodology
All arms use Grok 4.7 at high reasoning effort in an adapted Harvey harness. The study checks expert rubric criteria with an LLM judge and reports full-task success, mean rubric completion, tool failures, latency and cost. Vespper authored the study; Grok 4.7 also serves as judge.