[Paper Review] Enabling Conversational Interaction with Mobile UI using Large Language Models

Bryan Wang, Gang Li+1 more|arXiv (Cornell University)|Sep 18, 2022

AI in Service Interactions4 citations

TL;DR

This paper proposes a prompting-based approach that enables versatile conversational interaction with mobile UIs using a single large language model (LLM), eliminating the need for task-specific datasets and training. By converting Android UI view hierarchies into structured HTML text representations, the method achieves competitive performance on four key tasks—screen question-generation, summarization, question-answering, and instruction-to-action mapping—using only two few-shot examples per task.

ABSTRACT

Conversational agents show the promise to allow users to interact with mobile devices using language. However, to perform diverse UI tasks with natural language, developers typically need to create separate datasets and models for each specific task, which is expensive and effort-consuming. Recently, pre-trained large language models (LLMs) have been shown capable of generalizing to various downstream tasks when prompted with a handful of examples from the target task. This paper investigates the feasibility of enabling versatile conversational interactions with mobile UIs using a single LLM. We designed prompting techniques to adapt an LLM to mobile UIs. We experimented with four important modeling tasks that address various scenarios in conversational interaction. Our method achieved competitive performance on these challenging tasks without requiring dedicated datasets and training, offering a lightweight and generalizable approach to enable language-based mobile interaction.

Motivation & Objective

• Address the challenge of enabling diverse conversational interactions with mobile UIs using a single, generalizable model.
• Overcome the limitations of existing assistants that lack GUI understanding and fail to answer UI-specific questions.
• Reduce reliance on costly, task-specific datasets and training pipelines for language-based mobile interaction.
• Explore the feasibility of adapting pre-trained LLMs to mobile UI tasks through few-shot prompting.
• Establish a lightweight, generalizable framework for rapid prototyping of language-based mobile interactions.

Proposed method

• Convert Android UI view hierarchies into text-based HTML representations using depth-first search traversal to preserve structural and property information.
• Design task-specific prompting templates to guide the LLM in performing four core mobile UI interaction tasks: screen question-generation, summarization, question-answering, and instruction-to-action mapping.
• Utilize in-context few-shot learning with only two examples per task to adapt the LLM without fine-tuning.
• Leverage the LLM’s pre-trained knowledge and generalization ability to infer UI semantics and generate contextually appropriate responses.
• Open-source the HTML conversion algorithm and example prompts to enable reproducibility and future extension.
• Evaluate performance using standard metrics and human evaluation, comparing against existing models like Screen2Words.

Experimental results

Research questions

• RQ1 Can a single pre-trained LLM be effectively adapted to multiple mobile UI conversational interaction tasks using few-shot prompting?
• RQ2 How well can an LLM perform on novel tasks like screen question-generation and question-answering without dedicated training data?
• RQ3 To what extent does the HTML representation of UI hierarchies preserve the necessary structural and semantic information for LLMs to reason about mobile interfaces?
• RQ4 How does the performance of the few-shot LLM approach compare to dedicated models trained on large-scale datasets?
• RQ5 What are the limitations of relying solely on view hierarchy data, and how do visual or multimodal cues affect LLM reliability in mobile UI interaction?

Key findings

• The proposed prompting method achieved competitive performance on four challenging mobile UI tasks—screen question-generation, summarization, question-answering, and instruction-to-action mapping—using only two few-shot examples per task.
• The method set benchmark performance for the novel task of screen question-generation, which had not been previously studied in the literature.
• Human evaluation showed that the LLM-generated screen summaries were more accurate than those from Screen2Words, a model trained on tens of thousands of examples.
• The LLM demonstrated emergent behaviors such as question combination and knowledge infusion, indicating strong reasoning and generalization capabilities.
• Despite the absence of visual input, the model performed well on text-based UI tasks, suggesting that view hierarchy-based HTML representation is effective for many interaction scenarios.
• The approach enables rapid prototyping of language-based mobile interactions, significantly reducing the time and cost of dataset collection and model training compared to traditional pipelines.