Benchmark
Android Bench
Evaluate coding agents on Android app development, migrations and architectural changes.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
Version 2.0 contains 30 long-horizon tasks spanning app creation, migrations, new features and app conversions. The earlier 1.0 benchmark evaluated localized pull requests.
Methodology
Version 2.0 combines multiple agent harnesses, multimodal UI verification with an LLM visual judge, and continuous completion scoring. Task design and trajectory audits address copied solutions and reward hacking.