[Paper Review] ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations

Yue Jiang, Eldon Schoop+2 more|arXiv (Cornell University)|Oct 7, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

This paper introduces ILuvUI, a vision-language model fine-tuned on 335K instruction-tuned, multimodal UI screenshots and text interactions generated via an LLM and UI element detector. The method eliminates human-annotated captions, enabling zero-shot UI understanding, and achieves strong performance on UI description, reasoning, and multi-step navigation tasks, outperforming non-UI-tuned VLMs.

ABSTRACT

Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data. In this paper, we adapt a recipe for generating paired text-image training data for VLMs to the UI domain by combining existing pixel-based methods with a Large Language Model (LLM). Unlike prior art, our method requires no human-provided annotations, and it can be applied to any dataset of UI screenshots. We generate a dataset of 335K conversational examples paired with UIs that cover Q&A, UI descriptions, and planning, and use it to fine-tune a conversational VLM for UI tasks. To assess the performance of our model, we benchmark it on UI element detection tasks, evaluate response quality, and showcase its applicability to multi-step UI navigation and planning.

Motivation & Objective

• To address the lack of multimodal training data for vision-language models (VLMs) in the UI domain.
• To develop a scalable, automated method for generating diverse, high-quality text-image pairs for UIs without human-annotation.
• To fine-tune a VLM for instruction-following in UI tasks, including description, reasoning, and multi-step navigation.
• To evaluate the model’s performance on UI understanding benchmarks and demonstrate its generalization to unseen UI tasks.

Proposed method

• Leverage an existing UI element detection model to extract structured representations from UI screenshots.
• Use a large language model (GPT-3.5-turbo) to generate six types of text responses—Q&A, descriptions, action lists, outcome predictions, element selection, and planning—based on the detected UI elements and context.
• Combine pixel-based UI screenshots with LLM-generated text to create 335K paired image-instruction examples for fine-tuning.
• Fine-tune the open-source LLaVA VLM on the synthetic UI dataset to create an instruction-following visual agent, ILuvUI.
• Apply prompt engineering with diverse templates to generate varied, realistic, and task-specific responses for each UI screenshot.
• Use the model for zero-shot and few-shot evaluation on UI element detection, description, and multi-step navigation tasks.

Experimental results

Research questions

• RQ1 Can a vision-language model be effectively fine-tuned on synthetic, instruction-tuned UI data without human-annotated captions?
• RQ2 How well does the resulting model perform on UI-specific tasks such as element detection, description, and action planning?
• RQ3 Can the model generalize to unseen UI layouts and perform multi-step navigation based on natural language instructions?
• RQ4 How does the model’s performance compare to non-UI-tuned VLMs on UI understanding benchmarks?

Key findings

• The ILuvUI model achieves strong performance on UI element detection and type identification, outperforming non-UI-tuned VLMs.
• The model generates high-quality, contextually relevant descriptions and action recommendations, as validated through qualitative and quantitative evaluation.
• ILuvUI demonstrates reasoning and planning capabilities, successfully navigating multi-step UI tasks such as starting a stopwatch through sequential action prediction.
• The synthetic data generation pipeline produces diverse, realistic, and task-specific responses without human-annotated captions, enabling scalable data creation.
• The model generalizes to unseen UI layouts and performs robustly across different application domains, as shown in benchmark evaluations.
• The approach enables zero-shot UI understanding and paves the way for accessible, voice-controlled UI interaction and automated GUI testing.