[Paper Review] Android in the Wild: A Large-Scale Dataset for Android Device Control

C. Rawles, Alice Li+3 more|arXiv (Cornell University)|Jul 19, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces Android in the Wild (AitW), a large-scale dataset of 715k human-annotated device interaction episodes across 30k unique natural language instructions, four Android versions (v10–13), and eight device types. The dataset enables training and evaluation of device-control agents that map natural language to precise gestures on real UIs, with a focus on robustness to new tasks, apps, and platform versions, demonstrating the feasibility of end-to-end, vision-based agent learning without reliance on UI metadata.

ABSTRACT

There is a growing interest in device-control systems that can interpret human natural language instructions and execute them on a digital device by directly controlling its user interface. We present a dataset for device-control research, Android in the Wild (AITW), which is orders of magnitude larger than current datasets. The dataset contains human demonstrations of device interactions, including the screens and actions, and corresponding natural language instructions. It consists of 715k episodes spanning 30k unique instructions, four versions of Android (v10-13),and eight device types (Pixel 2 XL to Pixel 6) with varying screen resolutions. It contains multi-step tasks that require semantic understanding of language and visual context. This dataset poses a new challenge: actions available through the user interface must be inferred from their visual appearance. And, instead of simple UI element-based actions, the action space consists of precise gestures (e.g., horizontal scrolls to operate carousel widgets). We organize our dataset to encourage robustness analysis of device-control systems, i.e., how well a system performs in the presence of new task descriptions, new applications, or new platform versions. We develop two agents and report performance across the dataset. The dataset is available at https://github.com/google-research/google-research/tree/master/android_in_the_wild.

Motivation & Objective

• To address the lack of large, diverse, and realistic datasets for training and evaluating device-control agents that operate directly on mobile UIs without relying on platform-specific UI metadata.
• To enable research into generalization across new task instructions, new applications, and evolving Android platform versions (v10–13) and device types (Pixel 2 XL to Pixel 6).
• To support the development of vision-based agents that map natural language commands to precise, coordinate-level gestures (e.g., swipes, taps) rather than abstract UI element actions.
• To provide a benchmark for evaluating agent performance under distribution shift, including unseen tasks, apps, and UIs, using a structured evaluation protocol with unseen domain splits.
• To facilitate the development of multimodal, foundation-model-based agents that can process raw screen pixels and generate low-level gesture actions.

Proposed method

• The dataset was collected via crowdsourced raters who executed natural language instructions on real Android devices, capturing full interaction trajectories with screenshots and precise <x,y> gesture actions.
• Raters used a mouse and keyboard to simulate touch interactions, recording taps, drags, and system button presses (e.g., home, back) with spatial precision.
• Hindsight relabeling was applied to multi-step episodes to generate single-step tasks, increasing data efficiency and enabling evaluation of goal-reaching performance.
• The dataset includes diverse task instructions sourced from human raters, LLM-generated prompts, and technical documentation (e.g., PixelHelp), ensuring linguistic and functional variety.
• The data spans 350+ Android apps and websites, with variations in screen resolution, device form factor, and Android version, enabling robustness evaluation across real-world distribution shifts.
• Evaluation is conducted using AndroidEnv, an open-source Android emulator platform, allowing reproducible benchmarking of agent performance under unseen conditions.

Figure 1: AitW data pipeline . Raters are given a randomly selected instruction. The raters execute the task by interacting with the device in a natural way. We capture precise gestures in addition to typing and the home and back button interactions (we plot swipes with the arrow pointing where the

Experimental results

Research questions

• RQ1 How well can device-control agents generalize to new task instructions not seen during training, especially when instructions are high-level or ambiguous?
• RQ2 To what extent can agents trained on AitW perform reliably across new Android platform versions (v10–13) and diverse device types (e.g., Pixel 2 XL to Pixel 6)?
• RQ3 How effective are vision-based agents in learning to map raw screen pixels and natural language to precise, coordinate-level gestures without access to structured UI metadata?
• RQ4 Can automated evaluation metrics (partial and complete match) reliably approximate human-annotated performance in zero-shot generalization settings?
• RQ5 What are the limitations of current imitation learning and LLM-based agents when applied to real-world, visually grounded, multi-step device control tasks?

Key findings

• The AitW dataset contains 715k episodes across 30k unique instructions, 350+ apps and websites, four Android versions (v10–13), and eight device types, making it orders of magnitude larger than prior datasets.
• Partial match correlation with human-annotated complete match is strong, especially for short tasks, validating automated metrics as a reliable proxy for true performance.
• The complete match metric is a lower bound estimate of true performance, indicating that automated evaluation may underestimate agent success but remains useful for benchmarking.
• Two agents were trained on AitW: a behavior cloning model and an LLM-based agent, with the former outperforming the latter, suggesting that dedicated models may still be more effective than fine-tuned foundation models for this task.
• The dataset enables robustness evaluation across unseen domains, including new tasks, new apps, and new Android versions, with a dedicated split for such generalization testing.
• Despite limitations in rater demographics and input method (mouse/keyboard vs. touch), the dataset captures realistic, diverse interaction patterns and supports research into generalization and multimodal learning.

(a)