AndroidWorld
Test whether an agent can complete tasks in working Android apps by checking what actually changed on the device.
Explore the original evidence
Explore the original evaluation, its methods, and reported findings.
Open evalCoverage
The original suite has 116 programmable tasks across 20 Android apps. Chris Rawles led the project, with Yifan Chang and Sarah Clinckemaillie credited as equal contributors. The linked paper lists the full team.
Methodology
Each run can vary a task’s parameters. The evaluator checks app or device state to determine success, so an agent need not follow a single recorded action sequence. Task success rate (also labeled AndroidWorld_SR) is a metric of this suite. Results depend on task variations. The integrated MiniWoB++ web tasks and the separate AndroidControl demonstration dataset are different evaluation settings.