Benchmark
SnowPR
Evaluate coding agents on real changes to Snowflake’s large Java, C++ and Go codebases.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
65 merged pull requests across 11 product areas: 62 from Snowflake’s monorepo and three from a Go repository. Article authors: Soyoung Yoon, Harshn Pathak, Yuxiong He, Zhewei Yao, Sourabh Shirhatti and Qian Jiang.
Methodology
Hidden tests check new behavior and regressions after agents modify a pre-change checkout. The study runs three trials per task and distinguishes individual-trial success from solving a task at least once. Snowflake also supplies Cortex Code, a competing agent. Tasks are private, and results apply to this company’s environment.