[Paper Review] Can We Revitalize Interventional Healthcare with AI-XR Surgical Metaverses?

Adnan Qayyum, Muhammad Bilal+6 more|arXiv (Cornell University)|Mar 25, 2023

Artificial Intelligence in Healthcare and EducationMedicine3 citations

TL;DR

This paper proposes AI-XR surgical metaverses—immersive, AI-powered virtual environments for preoperative planning in interventional surgery—demonstrating that adversarial attacks can subtly manipulate incision point localization in virtual patient models, leading to unsafe surgical plans. The key contribution is exposing critical security vulnerabilities in AI-XR surgical metaverses, urging the development of robust, secure systems before clinical deployment.

ABSTRACT

Recent advancements in technology, particularly in machine learning (ML), deep learning (DL), and the metaverse, offer great potential for revolutionizing surgical science. The combination of artificial intelligence and extended reality (AI-XR) technologies has the potential to create a surgical metaverse, a virtual environment where surgeries can be planned and performed. This paper aims to provide insight into the various potential applications of an AI-XR surgical metaverse and the challenges that must be addressed to bring its full potential to fruition. It is important for the community to focus on these challenges to fully realize the potential of the AI-XR surgical metaverses. Furthermore, to emphasize the need for secure and robust AI-XR surgical metaverses and to demonstrate the real-world implications of security threats to the AI-XR surgical metaverses, we present a case study in which the ``an immersive surgical attack'' on incision point localization is performed in the context of preoperative planning in a surgical metaverse.

Motivation & Objective

• To investigate the transformative potential of AI and extended reality (XR) in creating surgical metaverses for interventional healthcare.
• To identify and analyze critical security vulnerabilities in AI-XR surgical metaverses, particularly in preoperative planning.
• To demonstrate the feasibility of adversarial attacks that manipulate incision point localization in virtual surgical environments.
• To emphasize the urgent need for secure and robust design in AI-XR surgical metaverses to prevent life-threatening errors.
• To guide future development by highlighting the balance between innovation and patient safety in immersive surgical technologies.

Proposed method

• Developed a virtual surgical environment using Unity 3D and C# to simulate interventional procedures with 3D patient models and surgical instruments.
• Utilized digital twins of patients to enable preoperative simulation and incision point validation in a mixed reality setting.
• Implemented an imperceptible adversarial attack by introducing minute, undetectable movements in the patient’s digital twin during incision selection.
• Conducted experiments using an Oculus Quest 2 VR headset to simulate real-time interaction with the surgical metaverse and validate attack impact.
• Mapped optimal incision points (e.g., ribs 3-4, 4-5, 5-6) and tracked their adversarial displacement to incorrect locations (e.g., 5-6, 7-8) under attack.
• Evaluated the consequences of manipulated incision points on tool access, visualization, and surgical feasibility in the virtual environment.

Figure 1: Illustration of various potential applications that will be featured in the AI-XR surgical metaverse.

Experimental results

Research questions

• RQ1 Can AI-XR surgical metaverses effectively enhance preoperative planning in interventional surgery?
• RQ2 How can adversarial attacks subtly manipulate incision point localization in immersive surgical simulations?
• RQ3 What are the real-world implications of undetected manipulation in virtual surgical planning for patient safety?
• RQ4 What security measures are necessary to ensure the reliability and integrity of AI-XR surgical metaverses?
• RQ5 How can surgical metaverses be made robust against adversarial attacks while maintaining clinical utility?

Key findings

• An imperceptible adversarial attack successfully shifted optimal incision points from ribs 3-4, 4-5, and 5-6 to 3-4, 5-6, and 7-8 in the virtual environment.
• The attack caused significant surgical planning risks, including obstructed tool access and compromised visualization due to misaligned incisions.
• The manipulation of incision points could lead to irreversible surgical complications, such as the need for open-heart surgery, if undetected.
• The attack was executed by subtly altering the digital twin’s position during user selection, making it undetectable to human experts.
• The study demonstrates that current AI-XR surgical metaverses are vulnerable to adversarial manipulation, threatening patient safety.
• The findings underscore the urgent need for robust security mechanisms in AI-XR surgical metaverses before clinical adoption.

Figure 2: Illustration of an attack on incision points localization for minimally invasive mitral valve repair (MIMVr) surgery.