# Android Bench 2.0 shows frontier coding agents still fail most multi-day Android tasks Google has rebuilt Android Bench around multi-day engineering tasks, continuous completion scoring and provider-agent harnesses. The best model passes only about 28% of the new long-horizon tasks, versus roughly 91% on the benchmark’s earlier incremental work. Android Bench 2.0 moves coding-agent evaluation away from small repository fixes toward dependency upgrades, app builds, migrations and other jobs that can take a human engineer days. The results expose a much larger reliability gap than short-task benchmarks—and show that the agent harness can materially change cost and outcome. - Status: Active - Published: 2026-09-21T21:35:53+12:00 - Updated: 2026-09-21T21:35:53+12:00 - Categories: Artificial Intelligence, Web Development, AI Coding, Developer Tools - Tags: AI evaluations, Android, Coding agents - Canonical HTML: https://beyondthe.news/dossiers/android-bench-2-long-horizon-agentic-coding-evaluation ## What changed On September 17, Google released Android Bench 2.0 with its first long-horizon tasks, agentic evaluation and continuous scoring. The new tasks include dependency upgrades, substantial features, apps built from scratch and cross-platform-to-Android conversions that Google says can take an engineer multiple days or a week. Instead of only binary pass/fail grading, the benchmark now reports completion based on functionality, visual fidelity, regressions and structural constraints, alongside pass rate and average task cost. Google is also pairing models with provider agents such as Codex and Antigravity so the harness becomes part of the evaluation. ## Why it matters Coding-agent benchmarks have often rewarded bounded repository tasks that are much cleaner than the work developers increasingly delegate to agents. Android Bench 2.0 makes that mismatch measurable: the highest pass rate on its new long-horizon set is around 28%, compared with roughly 91% on the original tasks. For builders choosing coding models, that argues for evaluating whole task-and-harness combinations on migration, architecture and runtime-validation work rather than extrapolating from short-task benchmark scores. ## The benchmark now asks agents to do multi-day engineering work The long-horizon set includes dependency upgrades, feature implementation, building applications from scratch and porting cross-platform applications to Android. Google says these tasks reflect work that can take an engineer several days or a week, rather than the bug fixes and smaller feature requests common in the original benchmark. ## Pass/fail was hiding useful partial completion Google replaced binary-only interpretation with a continuous completion score that combines functionality, visual fidelity and regression avoidance, with penalties for violating instructions or structural constraints. A run that correctly refactors most of an application but misses one edge-case assertion can therefore be distinguished from a run that barely starts the task. The leaderboard still exposes pass rate as a stricter measure alongside completion and cost. ## The reliability gap gets much larger on long work Google reports that the best pass rate on the new long-horizon tasks is around 28%, versus roughly 91% on the original Android Bench tasks. The tested systems are better at writing new code and deterministic transformations than at refactors, migrations and work requiring runtime validation. Google says no model fully solves the cross-platform-porting challenge, with frontier systems reaching at most about 80% completion. ## The harness is becoming part of the model decision Android Bench 2.0 adds agent evaluations using provider-associated harnesses—for example, Google says it ran GPT-5.6 Sol through Codex and Gemini 3.8 Flash through Antigravity. Google specifically notes that harness techniques such as prompt caching and compact tool windows can reduce token use. Future leaderboard work is intended to compare more model-and-agent combinations, making the practical unit of evaluation the system rather than the checkpoint alone. ## Key details - Google released Android Bench 2.0 on September 17, 2026. - Its first long-horizon tasks are designed around work that can take human engineers multiple days or a week. - Tasks include dependency upgrades, new features, greenfield app builds and cross-platform-to-Android conversions. - Google now reports continuous completion scoring as well as strict pass rate and average task cost. - The highest reported long-horizon pass rate is around 28%, versus roughly 91% on the original benchmark tasks. - GPT-6 Astra leads the published long-horizon pass-rate table at 28% in Google’s September 17 snapshot. - Google is beginning to evaluate models inside provider agent harnesses rather than treating the model alone as the full coding system. ## Builder takeaways - Do not use a short-task coding benchmark as a proxy for whether an agent can own a migration or multi-day feature end to end. - Measure partial completion, regressions and runtime correctness separately; a binary success metric can hide both useful progress and dangerous near-success. - Benchmark the model and harness together. Tool-window design, caching, context management and agent scaffolding can affect cost and outcomes. - Include migration and refactor tasks in internal evaluations because Google’s results suggest they expose architectural and runtime weaknesses that greenfield code generation does not. - Keep human checkpoints around long-horizon work: even the leading published system still fails the strict pass criterion on most tasks. ## What to watch - Google’s expansion of the public long-horizon task set and methodology. - Cross-harness comparisons using the same model, which would isolate how much agent scaffolding contributes. - Independent reproduction of Android Bench 2.0 results and sensitivity to task contamination. - Whether long-horizon pass rates rise quickly enough to change review and delegation practices for production teams. - Cost-per-completed-task results as providers optimize models and agent harnesses together. ## Uncertainties - Android Bench is maintained by Google, which also develops Gemini and Antigravity; its methodology is public but the benchmark operator is not vendor-neutral. - The September 17 long-horizon set is an initial release and scores may move as tasks, models and harnesses expand. - Provider-associated agent pairings do not yet isolate model quality from harness quality. - Android-specific results should not be assumed to transfer directly to every language, framework or repository type. ## Sources - [Android Bench 2.0: Pushing the frontier with challenging long-horizon tasks](https://android-developers.googleblog.com/2026/09/android-bench-2-long-horizon-tasks.html) — Google / Android Developers · primary · 2026-09-17T00:00:00+12:00. Primary announcement describing long-horizon tasks, continuous scoring, agent evaluations, published results and limitations.