[Paper Review] AI-Enhanced Virtual Reality in Medicine: A Comprehensive Survey

Yixuan Wu, Kaiyuan Hu+2 more|arXiv (Cornell University)|Feb 5, 2024

COVID-19 diagnosis using AI4 citations

TL;DR

This paper presents the first systematic survey of AI-enhanced virtual reality (VR) in medicine, introducing a three-tiered taxonomy—Visualization Enhancement, VR-related Medical Data Processing, and VR-assisted Intervention—to categorize and analyze AI-VR applications across medical diagnosis, treatment, and training. It highlights how AI-driven VR improves diagnostic accuracy, surgical planning, and remote collaboration, while identifying key challenges and future directions in data integration, ethics, and clinical adoption.

ABSTRACT

With the rapid advance of computer graphics and artificial intelligence technologies, the ways we interact with the world have undergone a transformative shift. Virtual Reality (VR) technology, aided by artificial intelligence (AI), has emerged as a dominant interaction media in multiple application areas, thanks to its advantage of providing users with immersive experiences. Among those applications, medicine is considered one of the most promising areas. In this paper, we present a comprehensive examination of the burgeoning field of AI-enhanced VR applications in medical care and services. By introducing a systematic taxonomy, we meticulously classify the pertinent techniques and applications into three well-defined categories based on different phases of medical diagnosis and treatment: Visualization Enhancement, VR-related Medical Data Processing, and VR-assisted Intervention. This categorization enables a structured exploration of the diverse roles that AI-powered VR plays in the medical domain, providing a framework for a more comprehensive understanding and evaluation of these technologies. To our best knowledge, this is the first systematic survey of AI-powered VR systems in medical settings, laying a foundation for future research in this interdisciplinary domain.

Motivation & Objective

• To establish a comprehensive, systematic framework for classifying AI-enhanced VR applications in medicine.
• To address the lack of structured overviews in the interdisciplinary domain of AI and VR for healthcare.
• To analyze the technical, clinical, and ethical challenges in deploying AI-VR systems in real-world medical settings.
• To identify research gaps and future trajectories in immersive medical technologies, particularly in diagnostics, intervention, and telemedicine.
• To support the development of trustworthy, user-accepted, and clinically viable AI-VR systems through evidence-based evaluation of current practices.

Proposed method

• Proposes a novel three-category taxonomy: Visualization Enhancement, VR-related Medical Data Processing, and VR-assisted Intervention.
• Reviews advanced techniques such as generative models, neural implicit functions, and scene graph modeling for reconstructing and interpreting medical VR environments.
• Analyzes AI-driven visual question answering (VQA) and visual question localized-answering (VQLA) for real-time surgical context understanding.
• Evaluates the role of multimodal datasets like MVOR, 4D-OR, and OR phase recognition datasets in enabling AI training and surgical understanding.
• Integrates natural language processing (NLP) and sonification for enhanced clinical interpretation and patient-VR interaction.
• Examines human-in-the-loop collaboration systems using interactive AI to support real-time decision-making in surgical and diagnostic workflows.

Figure 1: Demonstrating the workflow of AI-enhanced VR-assisted medical services. First, Visualization Enhancement focuses on the augmentation of a user’s visual and sensory perception within a virtual space. Next, VR-related Medical Data Processing discusses how VR, combined with AI, facilitates in

Experimental results

Research questions

• RQ1 How can AI-enhanced VR be systematically categorized across the medical care lifecycle to improve clinical understanding and application?
• RQ2 What are the key technical enablers—such as generative models, scene graphs, and VQA—enabling accurate and immersive medical VR experiences?
• RQ3 How do AI-VR systems support real-time procedural guidance, surgical planning, and interdisciplinary collaboration in complex medical environments?
• RQ4 What are the major data, ethical, and usability challenges hindering the adoption of AI-VR in clinical practice?
• RQ5 What future technological and clinical advancements can be expected in AI-VR for telemedicine, mental health, and personalized diagnostics?

Key findings

• The proposed taxonomy of Visualization Enhancement, VR-related Medical Data Processing, and VR-assisted Intervention provides a structured framework for understanding AI-VR applications in medicine.
• Generative models and neural implicit functions enable high-fidelity reconstruction of anatomical structures from limited medical imaging data, improving VR visualization quality.
• VQA and VQLA systems significantly enhance surgical context understanding by enabling AI to answer complex questions and localize answers in real-time intraoperative scenes.
• Scene graph modeling offers a semantically rich, human-readable representation of surgical environments, capturing tools, anatomy, and interactions for improved procedural analysis.
• Public datasets like 4D-OR and MVOR are critical for training AI models in surgical phase recognition, motion analysis, and semantic scene understanding.
• Despite progress, challenges in data scarcity, ethical compliance, system transparency, and user trust remain major barriers to clinical deployment.