Benchmark
PostHog ReviewHog Incident Evaluation
Test whether code-review agents detect bugs that caused real PostHog incidents.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
A 33-incident evaluation links bug-introducing pull requests to incident descriptions and fixes. Alex Lebedev describes the work for PostHog.
Methodology
Recreate isolated versions of the offending pull requests and compare whether review agents identify the incident-causing bugs. The study distinguishes this set from an earlier single-PR test and separate Scout rubric experiments. PostHog evaluates its own product; repository history and other reviews can leak answers.