AndroidInTheWild: A Large-Scale Dataset For Android Device Control
Dec 10, 2023
Speakers
Christopher Rawles
Speaker · 0 followers
Daniel Rodriguez
Speaker · 0 followers
Oriana Riva
Speaker · 0 followers
Timothy Lillicrap
Speaker · 8 followers
About
Recently, we have seen a growing interest in device-control systems that can interpret human natural language instructions and execute them on a device by directly controlling its UI. We present a new dataset for mobile device control, AndroidInTheWild, which is orders of magnitude larger than current datasets. The dataset contains human-collected demonstrations of natural language instructions, UI screens, and actions. It consists of 715k episodes spanning 30k unique prompts, four versions of Android (v10–13), and 8 different device types (Pixel 2 XL to Pixel 6) with varying screen resolutions. It contains long-horizon multi-step tasks that require semantic understanding. In addition to its language complexity, this dataset poses a new challenge as the UI elements composing an application's screen must be inferred solely from its visual appearance. Instead of simple UI element-based actions, the action space consists of precise gestures allowing for a higher-degree of interactivity (e.g., horizontal scrolls to operate carousel widgets). We organize our dataset to support and encourage robustness analysis of device-control systems, i.e., how well a system performs in the presence of new task descriptions, new applications or new platform versions. We develop two baseline agents and report performance on the dataset. The dataset is available at https://github.com/google-research/google-research/tree/master/android_in_the_wild.Recently, we have seen a growing interest in device-control systems that can interpret human natural language instructions and execute them on a device by directly controlling its UI. We present a new dataset for mobile device control, AndroidInTheWild, which is orders of magnitude larger than current datasets. The dataset contains human-collected demonstrations of natural language instructions, UI screens, and actions. It consists of 715k episodes spanning 30k unique prompts, four versions of …
Organizer
NeurIPS 2023
Account · 624 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
![[Re] Variational Neural Cellular Automata](https://ma.slideslive.com/library/presentations/39011085/thumbnail/re-variational-neural-cellular-automata_Fm4aa1_small.png)
[Re] Variational Neural Cellular Automata
Watch later
Simon Sondén, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Framework and Benchmarks for Combinatorial and Mixed-variable Bayesian Optimization
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Winning Submission Trojan Detection Track - Base Subtrack
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
VillanDiffusion: A Unified Backdoor Attack Framework for Diffusion Models
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%