[Paper Review] RescueAR: Augmented Reality Supported Collaboration for UAV Driven Emergency Response Systems

Ankit Agrawal, Jane Cleland‐Huang|arXiv (Cornell University)|Sep 30, 2021

Augmented Reality Applications4 citations

TL;DR

RescueAR is a location-based augmented reality platform that enables real-time geotagging, visualization, and sharing of emergency scene information among human responders and UAVs. Using three interfaces—Flying-AR, FirstResponse-AR, and Mission-Control—it improves spatial cognition and communication accuracy, achieving POI geolocation within 2 meters under ideal conditions.

ABSTRACT

Emergency response events are fast-paced, noisy, and they require teamwork to accomplish the mission. Furthermore, the increasing deployment of Unmanned Aerial Vehicles (UAVs) alongside emergency responders, demands a new form of partnership between humans and UAVs. Traditional radio-based information exchange between humans during an emergency response suffers from a lack of visualization and often results in miscommunication. This paper presents a novel collaboration platform: RescueAR, which utilizes the paradigm of Location-based Augmented Reality to geotag, share, and visualize information. RescueAR aims to support the two-way communication between humans and UAVs, facilitate collaboration across diverse responders, and visualize scene information relevant to the rescue team's role. According to our feasibility study, a user study, followed by a focus group session with police officers, RescueAR can support rescue teams in developing the spatial cognition of the scene, facilitate the exchange of geolocation information, and complement existing communication tools during the UAV-supported emergency response.

Motivation & Objective

• Address communication breakdowns in emergency response caused by noisy, non-visual radio communication.
• Improve spatial awareness and situational understanding among first responders and remote teams during UAV-supported missions.
• Enable two-way collaboration between humans and UAVs through geotagged, AR-visualized information sharing.
• Support diverse emergency response roles—on-scene, remote monitoring, and mission command—through role-specific AR interfaces.
• Evaluate the feasibility and usability of AR-based collaboration in real-world emergency scenarios using expert feedback and controlled studies.

Proposed method

• Design and implement RescueAR, a three-component AR system: Flying-AR (remote annotation of aerial video), FirstResponse-AR (on-scene AR visualization of geolocated points of interest), and Mission-Control (centralized spatial summary of UAVs, video streams, and POIs).
• Use GPS and RTK (Real-Time Kinematic) base stations to achieve centimeter-level geolocation accuracy for POIs in urban environments.
• Integrate live aerial video feeds with geotagging capabilities, allowing remote users to mark injured persons or hazards in real time.
• Enable AR visualization of geolocated POIs on mobile devices and AR glasses for on-site responders to perceive spatial context directly in their field of view.
• Conduct a usability study with six participants and a focus group with police officers to evaluate system performance and user experience.
• Analyze geolocation accuracy using pre-recorded video and known POI coordinates, achieving 2-meter precision under controlled conditions.

Experimental results

Research questions

• RQ1 Can AR-based geotagging and visualization improve situational awareness and spatial cognition during UAV-supported emergency response?
• RQ2 How do emergency responders perceive the usability and value of AR interfaces in sharing and interpreting geolocated information during high-stress scenarios?
• RQ3 What are the key design trade-offs in AR interface design when balancing real-time video flow with annotation precision during fast-paced missions?
• RQ4 To what extent can RescueAR reduce communication errors and improve coordination between on-site responders, remote monitors, and mission commanders?
• RQ5 How do varying levels of UAV experience among responders affect the adoption and effectiveness of AR-supported collaboration tools?

Key findings

• RescueAR achieved a geolocation accuracy of within 2 meters for points of interest under ideal conditions, demonstrating reliable spatial referencing.
• Users in the usability study reported that Flying-AR enhanced their ability to maintain consistent situational awareness of the evolving scene.
• Police officers in the focus group expressed strong support for RescueAR, indicating it could serve as a valuable complement to existing communication systems in law enforcement operations.
• Four out of six usability study participants preferred pausing the aerial video to ensure annotation precision, highlighting a critical UX trade-off between real-time flow and annotation accuracy.
• The system’s ability to visualize geolocated POIs in AR significantly improved the clarity and persistence of shared information compared to traditional radio-based communication.
• Participants identified the need for video buffering and adjustable playback speed to support annotation without losing situational context, suggesting future interface enhancements are essential.