[Paper Review] RealitySummary: Exploring On-Demand Mixed Reality Text Summarization and Question Answering using Large Language Models

Aditya Gunturu, Shivesh Jadon+5 more|arXiv (Cornell University)|May 28, 2024

Augmented Reality Applications4 citations

TL;DR

RealitySummary introduces an on-demand mixed reality reading assistant that uses Google Cloud OCR and GPT-4 to extract, summarize, and augment any printed or digital document in real time, projecting contextual summaries, keyword lists, timelines, and information cards via Microsoft HoloLens 2 and Apple Vision Pro. The system enables dynamic, AI-powered document enhancement without pre-processing, demonstrating strong usability and engagement in user studies.

ABSTRACT

Large Language Models (LLMs) are gaining popularity as reading and summarization aids. However, little is known about their potential benefits when integrated with mixed reality (MR) interfaces to support everyday reading. In this iterative investigation, we developed RealitySummary, an MR reading assistant that seamlessly integrates LLMs with always-on camera access, OCR-based text extraction, and augmented spatial and visual responses. Developed iteratively, RealitySummary evolved across three versions, each shaped by user feedback and reflective analysis: 1) a preliminary user study to understand reader perceptions (N=12), 2) an in-the-wild deployment to explore real-world usage (N=11), and 3) a diary study to capture insights from real-world work contexts (N=5). Our empirical studies' findings highlight the unique advantages of combining AI and MR, including always-on implicit assistance, long-term temporal history, minimal context switching, and spatial affordances, demonstrating significant potential for future LLM-MR interfaces beyond traditional screen-based interactions.

Motivation & Objective

• To develop a generalizable, on-demand mixed reality reading assistant that enhances any physical or digital document without requiring pre-processing.
• To identify and implement effective document enhancement techniques—such as summarization, comparison, and navigation—through a formative design study with six participants.
• To evaluate the usability and real-world applicability of AI-powered MR document augmentation through controlled and in-the-wild user studies.
• To explore the integration of multimodal inputs (e.g., voice, gaze) and future extensions like cross-page summarization and digital document linking.

Proposed method

• Utilizes Google Cloud OCR and OpenCV to extract text and images from physical or digital documents in real time.
• Employs GPT-4 to generate summaries, keyword lists, comparison tables, timelines, and information cards from extracted text.
• Uses markerless AR tracking based on document image features to spatially anchor and render augmented content around the document in mixed reality.
• Integrates Google’s speech-to-text API to enable voice-activated commands for summarization and querying.
• Projects generated content as a spatial 2D canvas around the document using HoloLens 2 and Apple Vision Pro.
• Combines OCR, LLMs, and AR rendering to enable real-time, context-aware document augmentation without prior document preparation.

Experimental results

Research questions

• RQ1 What are the most effective and generalizable techniques for enhancing diverse document types through mixed reality?
• RQ2 How do users interact with on-demand, AI-generated document augmentations in real-time mixed reality environments?
• RQ3 What are the technical and usability challenges in deploying OCR and LLMs for real-time, on-demand document enhancement in mixed reality?
• RQ4 How does multimodal input (e.g., voice, gaze) enhance the functionality and adaptability of AI-powered reading assistants?
• RQ5 What are the long-term usability and engagement prospects of such systems in real-world reading scenarios?

Key findings

• A formative design study with six participants identified five core categories of document enhancement—summarization, augmentation, navigation, comparison, and extraction—leading to the implementation of six key features in the system.
• The usability study (N=12) confirmed that participants valued the always-on, implicit interaction model and found the spatial, tangible nature of MR content significantly enhanced engagement and comprehension.
• In-the-wild study (N=11) demonstrated practical applicability across diverse real-world reading contexts, with users appreciating the system’s adaptability to unprepared documents.
• AR tracking instability was observed with text-only documents, especially in low-light conditions, highlighting limitations in current AR tracking robustness.
• OCR and image tracking performance varied with document quality, introducing processing delays and potential inaccuracies.
• LLM-generated content occasionally contained errors in ambiguous or highly specialized text, indicating a need for improved prompt engineering or domain fine-tuning.