Evolving how LLMs are measured for Android: the next era of Android Bench

Back in March, we introduced Android Bench—our LLM leaderboard for real-world Android development tasks. Our goal was to provide transparency around model capabilities in Android development and to encourage model improvements, to give you more helpful AI options for your everyday workflow. Since then, we have enhanced the benchmark based on your feedback, including evaluating open-weight models and adding cost and efficiency dimensions to the leaderboard.

But AI capabilities are ever-evolving, and measurement needs to follow suit. As part of our July release, we have adopted the Harbor framework, which includes an updated version of the benchmarking agent used to evaluate models.

Upgrading our methodology with the Harbor framework

When we designed Android Bench, we anchored our methodology on leading industry standards available at the time. We used mini-swe-agent v1, a general-purpose benchmarking agent, and adapted it to the nuances of Android development to provide a baseline measurement for the capabilities of models for common Android development tasks.

To continue providing you with state-of-the-art evaluations that accurately measure the latest model capabilities on Android development, we are standardizing our benchmark to the Harbor framework. Harbor defines standards and integrations that make it easy for anyone to run the benchmark, evaluate their preferred set-up, or share results – providing you with additional transparency and visibility.

This upgrade enables us to more rigorously evaluate models and their capabilities, and we re-ran the benchmark on all models to establish an updated baseline. This means there is a minor shift in scoring, but you will still be able to view historical scores within the archive on our website.

We want to ensure Android Bench is helpful for you, so we will continuously update it as our evaluations and the industry mature.

Expanding the leaderboard with 8 new models

As part of our commitment to keeping the leaderboard fresh, we have added Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, Qwen 3.7 Plus and Qwen 3.7 Max to the Android Bench leaderboard.

You will see that Claude Fable 5 is at the top of the leaderboard with a score of 84.5, followed by GPT 5.5 with 80.2, with Claude Sonnet 5 in 3rd with a score of 76.2.

When just comparing Open-weight models, GLM 5.2 is at the top with 72.2, followed by Kimi K2.7 Code with a score of 70.4.

You can check out model performance and efficiency metrics on the updated leaderboard to see how these new and previous models navigate Android-specific challenges like Jetpack Compose migrations, wearable networking, and platform API updates.

Opening Android Bench to community contributions

From the beginning, we’ve valued an open and transparent approach, which is why we made our original methodology and test harness publicly available on GitHub. You’ve asked for a way to provide feedback on our dataset, so now we’re taking collaboration a step further by giving you, the Android developer community, a chance to shape Android Bench.

Starting today, you can contribute to Android Bench in two ways:

• Design and submit your own Android development tasks to evaluate how models handle the scenarios that matter to you.


• Run and share benchmark evaluations firsthand, testing your preferred models against our dataset or your own custom tasks.

We will be reviewing the submitted tasks and will be assessing if they get added to the benchmark. We hope to build a benchmark that truly reflects the diverse, day-to-day realities of the global Android developer community.

Looking ahead

With more and more options for agentic development, maintaining a cutting-edge benchmark ensures that the AI assistance you rely on keeps getting smarter, more helpful, and more effective. Head over to our GitHub repository to check out the tasks. We invite you to submit a task to our team for review, and you can check out Harbor Hub to explore the dataset or submit evaluations.

As always, you can find the updated leaderboard, or read the methodology on our website.