[Paper Review] Demonstrating Performance Benefits of Human-Swarm Teaming

William Hunt, Jack Ryan+3 more|arXiv (Cornell University)|Mar 22, 2023

Simulation Techniques and ApplicationsDecision Sciences3 citations

TL;DR

This paper introduces HARIS, a web-based simulation platform enabling human-swarm teaming for search and rescue scenarios, where human operators augment autonomous UAV swarms by classifying targets earlier via real-time visual feedback. The system demonstrates that even untrained users significantly improve performance—measured by speed-accuracy—over fully autonomous swarms by reducing unnecessary missions through timely human intervention.

ABSTRACT

Autonomous swarms of robots can bring robustness, scalability and adaptability to safety-critical tasks such as search and rescue but their application is still very limited. Using semi-autonomous swarms with human control can bring robot swarms to real-world applications. Human operators can define goals for the swarm, monitor their performance and interfere with, or overrule, the decisions and behaviour. We present the ``Human And Robot Interactive Swarm'' simulator (HARIS) that allows multi-user interaction with a robot swarm and facilitates qualitative and quantitative user studies through simulation of robot swarms completing tasks, from package delivery to search and rescue, with varying levels of human control. In this demonstration, we showcase the simulator by using it to study the performance gain offered by maintaining a ``human-in-the-loop'' over a fully autonomous system as an example. This is illustrated in the context of search and rescue, with an autonomous allocation of resources to those in need.

Motivation & Objective

• To address the challenge of managing large robot swarms in safety-critical environments like disaster response, where full autonomy lacks reliability and full human control is infeasible.
• To develop a flexible, accessible simulation environment that supports multi-user interaction with robot swarms for human-swarm teaming research.
• To evaluate the performance benefits of human-in-the-loop control in dynamic, real-world-like scenarios such as search and rescue.
• To enable rapid deployment of custom scenarios using JSON-based configuration and integration with real-world maps via Google Maps.
• To facilitate user studies by providing a low-code, browser-accessible platform for testing human-robot interaction in swarm systems.

Proposed method

• HARIS is a Java-based web application built on the HutSim digital twin platform, enabling real-time simulation of multi-UAV swarms in human-supervised modes.
• The system uses the max-sum decentralized coordination algorithm for autonomous task allocation, minimizing total energy cost across UAVs.
• Human operators interact via a dynamic interface showing real-time UAV camera feeds with increasing resolution as agents approach targets.
• Operators can manually classify targets as 'Casualty' or 'No Casualty' when image clarity permits, reducing wasted missions.
• The interface supports customizable lenses and real-time scenario editing, allowing users to define new maps, agents, targets, and hazards using JSON configuration.
• Performance is evaluated using a combined metric of classifications per minute and accuracy, comparing human-swarm teams against fully autonomous swarms.

Figure 1. The HARIS interface uses Google Maps to provide a realistic command centre for the swarm

Experimental results

Research questions

• RQ1 Can untrained human operators improve the performance of autonomous UAV swarms in search and rescue by providing early classifications?
• RQ2 To what extent does human-in-the-loop intervention reduce unnecessary UAV missions compared to fully autonomous operation?
• RQ3 How does the integration of human reasoning enhance the speed-accuracy trade-off in dynamic, real-world-like scenarios?
• RQ4 Can a low-code, browser-based simulation platform like HARIS effectively support scalable, multi-user human-swarm teaming experiments?
• RQ5 What is the impact of human preferences and constraints on decentralized task allocation algorithms in swarm robotics?

Key findings

• Untrained human users significantly outperformed fully autonomous swarms in the speed-accuracy metric by classifying targets earlier, reducing unnecessary UAV missions.
• The human-swarm teaming mode reduced the number of wasted journeys, as operators could identify non-casualties before UAVs reached the target.
• The system demonstrated that even non-expert users could meaningfully enhance swarm performance through timely, low-effort interventions.
• HARIS enabled rapid scenario setup—within minutes—using real-world maps and JSON-based configuration, supporting immediate deployment for training and research.
• The simulation platform successfully supported multi-user interaction and real-time monitoring, validating its utility for large-scale user studies.
• The integration of human feedback into the max-sum algorithm via constraints improved decision-making without requiring full reconfiguration of the autonomous system.

Figure 2. Users are shown a simulated image of the target which gets clearer as the agent gets closer to the target. The user can classify the target early, saving the operation time.