[Paper Review] LLMR: Real-time Prompting of Interactive Worlds using Large Language Models

Fernanda De La Torre, Cathy Mengying Fang+4 more|arXiv (Cornell University)|Sep 21, 2023

Educational Games and Gamification10 citations

TL;DR

LLMR is a framework that real-time generates and edits interactive 3D scenes in Unity by orchestrating specialized GPT modules (Planner, Scene Analyzer, Builder, Inspector, Skill Library) with a runtime Roslyn compiler, achieving robust, real-time MR experiences.

ABSTRACT

We present Large Language Model for Mixed Reality (LLMR), a framework for the real-time creation and modification of interactive Mixed Reality experiences using LLMs. LLMR leverages novel strategies to tackle difficult cases where ideal training data is scarce, or where the design goal requires the synthesis of internal dynamics, intuitive analysis, or advanced interactivity. Our framework relies on text interaction and the Unity game engine. By incorporating techniques for scene understanding, task planning, self-debugging, and memory management, LLMR outperforms the standard GPT-4 by 4x in average error rate. We demonstrate LLMR's cross-platform interoperability with several example worlds, and evaluate it on a variety of creation and modification tasks to show that it can produce and edit diverse objects, tools, and scenes. Finally, we conducted a usability study (N=11) with a diverse set that revealed participants had positive experiences with the system and would use it again.

Motivation & Objective

• Motivate real-time generation and modification of interactive 3D scenes using LLMs in Unity-based mixed reality.
• Develop a modular LLM-driven pipeline that tackles data-scarce and complex scene synthesis tasks.
• Improve robustness and reduce code errors in generated Unity/C# scripts through iterative inspection and memory management.
• Demonstrate cross-platform interoperability and real-time execution of generated scenes.
• Provide usability evidence and open resources to encourage future work in AI-assisted MR scene authoring.

Proposed method

• Propose LLMR as an orchestration of specialized GPTs (Planner, Scene Analyzer, Builder, Inspector, Skill Library) to generate Unity C# code.
• Use a runtime Roslyn compiler to compile and execute generated scripts within Unity for real-time scene creation.
• Introduce the Builder-Inspector feedback loop to self-debug code before execution.
• Implement a Scene Analyzer to summarize scene state and condition code generation on a compact representation.
• Incorporate a Memory Management strategy with memory modes to balance token limits, performance, and interpretability.
• Evaluate with ablations, cross-platform demonstrations, and a user study (N=11) to assess usability and robustness.

Experimental results

Research questions

• RQ1 Can real-time prompt-driven generation produce functional, interactive 3D scenes in Unity using LLMs?
• RQ2 How do Planner, Scene Analyzer, Builder, Inspector, and Skill Library components interact to improve robustness and reduce errors?
• RQ3 What are the effects of memory management on token usage, performance, and interpretability in an LLM-driven MR pipeline?
• RQ4 To what extent can LLMR achieve cross-platform interoperability and real-time responsiveness across devices and plugins?
• RQ5 What is the user experience of non-expert participants using LLMR for scene creation and modification?

Key findings

• LLMR outperforms off-the-shelf GPT-4, with more than a 3x reduction in code errors in both empty and existing scenes.
• The system achieves completion times around one minute for task sequences.
• A structured modular pipeline with Planner, Scene Analyzer, Builder, and Inspector plus Skill Library improves robustness via a code-inspector feedback loop.
• A usability study with 11 participants reported intuitive use and willingness to reuse the system.
• Cross-platform demonstrations show interoperability with Unity, Sketchfab plugins, and external sensors, supporting versatile MR workflows.