Google Updates Android Bench With Harbor: Claude Fable 5 Leads the New AI Coding Leaderboard

Introduction

Google has refreshed its Android Bench coding leaderboard and moved the benchmark workflow to the Harbor framework. The update matters because Android Bench is not a general coding contest. It is designed to test how well large language models handle real Android development tasks, including codebase understanding, patch generation, Android-specific APIs, build systems, and test validation.

The new results create an interesting signal for developers. Claude Fable 5 now sits at the top of the updated leaderboard, followed by GPT 5.5, while Google's own Gemini models show a more mixed profile across accuracy, latency, and cost. For teams that use AI coding agents in Android projects, the main takeaway is not simply “which model won.” It is how the evaluation method, sandbox environment, cost, and latency together change the way model performance should be judged.

Google Moves Android Bench to the Harbor Framework

On July 9, Google announced a major update to Android Bench, its benchmark and leaderboard for AI-assisted Android development. The most important change is the adoption of Harbor, a standardized framework for running agent tasks in sandboxed environments.

Previously, Android Bench used an evaluation setup based on mini-swe-agent v1 and adapted it for Android development. In the new version, Google has shifted toward Harbor to make benchmark execution more standardized, isolated, and reproducible. The goal is to make it easier for developers and researchers to run independent evaluations, compare different agent setups, and share results in a more transparent way.

Google has also open-sourced the Android Bench tooling on GitHub. This gives the community a clearer view into how the benchmark works and opens the door for feedback, custom Android development tasks, and broader participation around model evaluation.

Why the Harbor Sandbox Change Matters

The framework behind a benchmark can affect the result. That is especially true for AI coding agents, where models do not just answer questions; they inspect repositories, run commands, edit files, call tools, and attempt to satisfy tests.

Harbor is built around sandboxed agent task evaluation. For Android Bench, that means model runs can be organized in a more controlled execution environment instead of relying on ad hoc local setups. This helps reduce ambiguity when comparing models and makes it easier to reproduce a given evaluation.

Google's Android Bench methodology also emphasizes Android-specific development problems rather than generic programming tasks. The benchmark includes work related to areas such as Jetpack Compose, Coroutines and Flows, Room, Hilt, navigation migration, Gradle configuration, SDK changes, media, camera, foldables, runtime permissions, and other common Android engineering concerns.

Updated Leaderboard Results

After the methodology change, Google re-ran the benchmark and refreshed the Android Bench leaderboard. According to the updated results, Claude Fable 5 ranks first with an 84.5% score. GPT 5.5 follows with 80.2%, and Claude Sonnet 5 ranks third with 76.2%.

A simplified view of the top results looks like this:

Rank	Model	Score	CI Range	Avg Latency	Avg Cost
1	Claude Fable 5	84.5%	79.9–88.8	8.0 h	$133.2
2	GPT 5.5	80.2%	73.5–86.6	11.4 h	$138.3
3	Claude Sonnet 5	76.2%	69.0–82.1	12.3 h	$99.9
4	GPT 5.4	74.1%	66.0–80.9	8.4 h	$83.4
5	Gemini 3.1 Pro Preview	73.7%	66.1–80.4	10.6 h	$87.4
6	Claude Opus 4.8	72.4%	65.8–79.3	6.7 h	$88.0
7	GLM 5.2	72.2%	65.3–78.7	38.9 h	$117.0
8	Gemini 3.5 Flash	71.1%	63.6–78.2	28.3 h	$165.6

These numbers should be read as benchmark-specific results. They do not prove that one model is universally better at all coding tasks. They show how each model performed under Google's updated Android-specific evaluation environment.

Gemini's Mixed Position in the New Results

The updated ranking is notable because Google's own models do not lead the chart. Gemini 3.1 Pro Preview ranks fifth with a score of 73.7%. Its reported average cost is lower than several top-ranked models, but its accuracy is behind Claude Fable 5, GPT 5.5, Claude Sonnet 5, and GPT 5.4.

Gemini 3.5 Flash is even more interesting. It is positioned as a lighter model, but in this benchmark it shows a long average latency of 28.3 hours and an average cost of $165.6 per full benchmark run. That makes it less attractive in this specific evaluation, despite the general expectation that lighter models should be faster and cheaper.

The larger lesson is simple: model selection for coding agents cannot be based only on brand, model family, or price per token. A model that appears cost-efficient in normal chat use may behave very differently when asked to solve 100 real Android development tasks across tool calls, repository edits, and test runs.

What Android Bench Actually Evaluates

Android Bench is designed to evaluate whether an LLM can act like a practical Android developer. It gives the model real-world issue descriptions and asks it to generate code changes that resolve the problem. The resulting patch is then checked against a validation setup.

Google's methodology says the benchmark includes 100 tasks selected from a much larger pool of pull requests. The selection focuses on Android repositories and real development workflows, including cases involving Kotlin, Java, Jetpack Compose, traditional Views, apps, libraries, small targeted changes, and larger code modifications.

This makes Android Bench different from simple code-completion tests. It is closer to an agentic software engineering evaluation, where the model needs to understand the repo, make an appropriate change, and survive automated verification.

Why This Matters for AI Coding Agents

AI coding workflows are moving from prompt-based code snippets toward autonomous or semi-autonomous agents. In a real Android project, an agent must navigate project structure, understand build constraints, edit multiple files safely, handle API changes, and run tests without breaking existing behavior.

A benchmark like Android Bench helps developers evaluate models against this kind of workflow. It also makes cost and latency visible. For production use, the best model is not always the one with the highest score. A team may prefer a slightly lower score if the model is much faster, cheaper, or more stable inside its own development environment.

The update also reinforces a broader point: benchmark methodology should evolve as AI agents evolve. Tool calling, sandboxing, execution traces, reproducibility, and cost tracking are now part of the evaluation, not optional extras.

FAQ

What is Android Bench?

Android Bench is Google's benchmark for evaluating large language models on Android development tasks. It focuses on real Android engineering problems rather than general coding trivia or isolated algorithm questions.

Why did Google move Android Bench to Harbor?

Google moved Android Bench to the Harbor framework to standardize benchmark execution in sandboxed environments. This helps make evaluations easier to reproduce, compare, and share across different model and agent setups.

Which model ranks first in the updated Android Bench leaderboard?

In the updated leaderboard described by the source article and Google's Android Bench page, Claude Fable 5 ranks first with an 84.5% score. GPT 5.5 follows with 80.2%, while Claude Sonnet 5 ranks third.

Does this mean Claude Fable 5 is the best coding model overall?

Not necessarily. Android Bench measures Android-specific coding performance under a defined benchmark setup. A model that performs best here may not be the best choice for web development, data engineering, DevOps, or other software tasks.

Why does latency matter in an AI coding benchmark?

Latency shows how long a model takes to complete the benchmark tasks. For real engineering teams, a high score is useful only if the model can complete work within an acceptable time window.

Why does benchmark cost matter?

Cost matters because agentic coding can involve many tool calls, long context windows, repository inspection, and repeated test runs. A model with strong accuracy but very high cost may be less practical for frequent production use.

Can developers run Android Bench themselves?

The Android Bench GitHub repository provides tooling and setup instructions for evaluating models. The official documentation also points developers to the dataset and methodology so they can better understand or reproduce the benchmark.

Related Tools

• Android Bench: Google's leaderboard for evaluating LLMs on Android development tasks.
• Android Bench GitHub Repository: The open-source framework and tooling for Android Bench evaluation.
• Harbor: A framework for specifying and evaluating sandboxed agent tasks.
• mini-swe-agent: A lightweight software engineering agent used in benchmark and coding-agent workflows.
• LiteLLM: A unified interface for calling many LLM providers using OpenAI-compatible formats.
• Android Studio: The official IDE for Android app development and testing.

Related Links

• Android Bench Leaderboard: Official Android Bench ranking page with score, cost, and latency dimensions.
• Android Developers Blog: Evolving How LLMs Are Measured for Android: Google's official post explaining the Harbor migration and leaderboard refresh.
• Android Bench Methodology: Official methodology page describing task selection, execution, and evaluation dimensions.
• Android Bench GitHub Repository: Official code and documentation for the benchmark framework.
• Harbor Framework: Official Harbor website for sandboxed agent evaluation.
• Jetpack Compose Documentation: Official Android UI toolkit documentation relevant to many Android development tasks.

Summary

Google's Android Bench update is more than a leaderboard refresh. By moving to the Harbor framework, the benchmark now places stronger emphasis on reproducible, sandboxed, Android-specific agent evaluation.

The new ranking puts Claude Fable 5 at the top, followed by GPT 5.5 and Claude Sonnet 5. Gemini remains competitive but does not lead the updated chart, and Gemini 3.5 Flash shows that lighter models are not automatically faster or cheaper in agentic coding evaluations.

For developers and AI teams, the useful takeaway is to compare accuracy, latency, cost, and benchmark methodology together. Android Bench is best read as a practical signal for Android coding agents, not as a universal ranking of all AI models.