Android Bench 2.0 shows frontier coding agents still fail most multi-day Android tasks

On September 17, Google released Android Bench 2.0 with its first long-horizon tasks, agentic evaluation and continuous scoring. The new tasks include dependency upgrades, substantial features, apps built from scratch and cross-platform-to-Android conversions that Google says can take an engineer multiple days or a week. Instead of only binary pass/fail grading, the benchmark now reports completion based on functionality, visual fidelity, regressions and structural constraints, alongside pass rate and average task cost. Google is also pairing models with provider agents such as Codex and Antigravity so the harness becomes part of the evaluation.

Coding-agent benchmarks have often rewarded bounded repository tasks that are much cleaner than the work developers increasingly delegate to agents. Android Bench 2.0 makes that mismatch measurable: the highest pass rate on its new long-horizon set is around 28%, compared with roughly 91% on the original tasks. For builders choosing coding models, that argues for evaluating whole task-and-harness combinations on migration, architecture and runtime-validation work rather than extrapolating from short-task benchmark scores.

The long-horizon set includes dependency upgrades, feature implementation, building applications from scratch and porting cross-platform applications to Android. Google says these tasks reflect work that can take an engineer several days or a week, rather than the bug fixes and smaller feature requests common in the original benchmark.

Google replaced binary-only interpretation with a continuous completion score that combines functionality, visual fidelity and regression avoidance, with penalties for violating instructions or structural constraints. A run that correctly refactors most of an application but misses one edge-case assertion can therefore be distinguished from a run that barely starts the task. The leaderboard still exposes pass rate as a stricter measure alongside completion and cost.

Google reports that the best pass rate on the new long-horizon tasks is around 28%, versus roughly 91% on the original Android Bench tasks. The tested systems are better at writing new code and deterministic transformations than at refactors, migrations and work requiring runtime validation. Google says no model fully solves the cross-platform-porting challenge, with frontier systems reaching at most about 80% completion.

Android Bench 2.0 adds agent evaluations using provider-associated harnesses—for example, Google says it ran GPT-5.6 Sol through Codex and Gemini 3.8 Flash through Antigravity. Google specifically notes that harness techniques such as prompt caching and compact tool windows can reduce token use. Future leaderboard work is intended to compare more model-and-agent combinations, making the practical unit of evaluation the system rather than the checkpoint alone.