[论文解读] Automating UI Optimization through Multi-Agentic Reasoning

Zhipeng Li, Christoph Gebhardt+2 more|arXiv (Cornell University)|Feb 13, 2026

Interactive and Immersive Displays被引用 0

一句话总结

AutoOptimization 是一个框架，利用顺序的视觉-语言模型代理，根据混合现实/增强现实环境中的口头用户指令，自动配置、优化和验证多目标 UI 布局。

ABSTRACT

We present AutoOptimization, a novel multi-objective optimization framework for adapting user interfaces. From a user's verbal preferences for changing a UI, our framework guides a prioritization-based Pareto frontier search over candidate layouts. It selects suitable objective functions for UI placement while simultaneously parameterizing them according to the user's instructions to define the optimization problem. A solver then generates a series of optimal UI layouts, which our framework validates against the user's instructions to adapt the UI with the final solution. Our approach thus overcomes the previous need for manual inspection of layouts and the use of population averages for objective parameters. We integrate multiple agents sequentially within our framework, enabling the system to leverage their reasoning capabilities to interpret user preferences, configure the optimization problem, and validate optimization outcomes.

研究动机与目标

• Automate the end-to-end UI adaptation workflow from user instructions to final layout.
• Enable dynamic, user-specific customization of multi-objective optimization for UI placement.
• Reduce reliance on manual setup and population-average parameters in UI optimization.
• Utilize vision-language model agents to interpret instructions, configure optimization, and validate results.

提出的方法

• Introduce a sequential, agent-based AutoOptimization framework integrating a vision-language model (VLM) and an optimization module.
• Use Ambiguity Detection to clarify user instructions before optimization.
• Translate user instructions into a dynamically configured multi-objective optimization problem with selectable objectives and parameters.
• Generate a Pareto front of candidate layouts via an optimization solver.
• Validate and select the final layout by comparing Pareto candidates to the user’s original instructions using a VLM.
• Aggregate user preferences over time to refine future instructions and outputs.

实验结果

研究问题

• RQ1 Can a VLM-based agent automate ambiguity detection and problem configuration for UI optimization from verbal instructions?
• RQ2 How effectively can a solver generate Pareto-optimal UI layouts that align with user instructions in MR/MR settings?
• RQ3 Can a VLM-based validation agent accurately select the final layout that best matches user intent from Pareto-optimal designs?
• RQ4 Does aggregating user preferences over time improve alignment between optimized layouts and user needs?

主要发现

• Ambiguity detection achieved about 91% accuracy in leave-one-user-out and 93% in leave-one-scenario-out cross-validation for MR instructions.
• VLM-based selection of layouts from Pareto-optimal candidates aligned with participant choices in studies (26 participants, 26 VLM instances, 72 candidates, 18 scenarios).
• End-to-end evaluation showed AutoOptimization layouts aligned more closely with user preferences and required fewer adjustments than baselines, with comparable user satisfaction to manual placement and reduced effort.
• Demonstrated effectiveness of AutoOptimization over baseline approaches in a mixed reality UI layout use case.