[Paper Review] Principles and Guidelines for Evaluating Social Robot Navigation Algorithms

Anthony Francis, Claudia Pérez-D’Arpino+29 more|arXiv (Cornell University)|Jun 29, 2023

Social Robot Interaction and HRI15 citations

TL;DR

The paper defines eight core principles for social robot navigation and presents guidelines, benchmarks, datasets, simulators, and a unified API to enable fair, repeatable Evaluation across methods and platforms.

ABSTRACT

A major challenge to deploying robots widely is navigation in human-populated environments, commonly referred to as social robot navigation. While the field of social navigation has advanced tremendously in recent years, the fair evaluation of algorithms that tackle social navigation remains hard because it involves not just robotic agents moving in static environments but also dynamic human agents and their perceptions of the appropriateness of robot behavior. In contrast, clear, repeatable, and accessible benchmarks have accelerated progress in fields like computer vision, natural language processing and traditional robot navigation by enabling researchers to fairly compare algorithms, revealing limitations of existing solutions and illuminating promising new directions. We believe the same approach can benefit social navigation. In this paper, we pave the road towards common, widely accessible, and repeatable benchmarking criteria to evaluate social robot navigation. Our contributions include (a) a definition of a socially navigating robot as one that respects the principles of safety, comfort, legibility, politeness, social competency, agent understanding, proactivity, and responsiveness to context, (b) guidelines for the use of metrics, development of scenarios, benchmarks, datasets, and simulators to evaluate social navigation, and (c) a design of a social navigation metrics framework to make it easier to compare results from different simulators, robots and datasets.

Motivation & Objective

• Define a crisp, actionable definition of social robot navigation grounded in safety, comfort, legibility, politeness, social competency, agent understanding, proactivity, and contextual appropriateness.
• Provide guidelines for metrics, scenario design, benchmarks, datasets, and simulators to enable fair, repeatable evaluations.
• Develop a social navigation metrics framework to facilitate cross-simulator and cross-dataset comparisons.
• Review methodological lifecycle and propose a common API to unify simulator outputs for evaluation.

Proposed method

• Propose eight guiding principles (P1 safety, P2 comfort, P3 legibility, P4 politeness, P5 social competency, P6 understanding other agents, P7 proactivity, P8 contextual appropriateness) for social navigation.
• Offer a taxonomy and structured guidelines linking principles to experimental design, metrics, scenarios, benchmarks, datasets, and simulators.
• Discuss subjective human evaluation metrics, analytic metrics, and learned metrics within a unified evaluation framework.
• Outline a lifecycle for social navigation benchmarks, datasets, simulators, and deployments to support repeatable research.

Figure 1 : We identify eight broad principles of social robot navigation - including safety, comfort, legibility, politeness, social competency, agent understanding, proactivity, and contextual appropriateness - which motivate specific guidelines for experiments, metrics, scenarios, benchmarks, data

Experimental results

Research questions

• RQ1 How should social robot navigation be defined and decomposed into measurable principles?
• RQ2 What metrics, scenarios, benchmarks, datasets, and simulators are needed to compare social navigation algorithms fairly?
• RQ3 How can we design benchmarks and a unified API to enable cross-simulator and cross-dataset evaluations?
• RQ4 How should context influence the weighting of navigation principles in evaluation?
• RQ5 What is a practical lifecycle of data collection, issue discovery, and experimentation for advancing social navigation research?

Key findings

• Eight principles are proposed to guide social navigation: safety, comfort, legibility, politeness, social competency, agent understanding, proactivity, and contextual appropriateness.
• A taxonomy and guidelines link principles to experimental components (metrics, scenarios, benchmarks, datasets, simulators) to improve comparability.
• A social navigation metrics framework is proposed to enable cross-simulator and cross-dataset comparisons.
• A lifecycle for social navigation benchmarking is described, including data collection, issue discovery, laboratory experiments, and scenario development.
• The paper advocates a mix of real-world human–robot interaction studies and simulation-based ablation studies to build principled evaluations.

Figure 2 : We define a socially navigating robot as one that interacts with humans and other robots in a way that achieves its navigation goals while enabling other agents to achieve theirs. To make this objective achievable, we propose eight principles for social robot navigation: safety, comfort,