[论文解读] Principles and Guidelines for Evaluating Social Robot Navigation Algorithms

Anthony Francis, Claudia Pérez-D’Arpino+29 more|arXiv (Cornell University)|Jun 29, 2023

Social Robot Interaction and HRI被引用 15

一句话总结

本文定义了社交通机器人导航的八条核心原则，并提出指南、基准、数据集、仿真器，以及一个统一的 API，以在方法与平台之间实现公平、可重复的评估。

ABSTRACT

A major challenge to deploying robots widely is navigation in human-populated environments, commonly referred to as social robot navigation. While the field of social navigation has advanced tremendously in recent years, the fair evaluation of algorithms that tackle social navigation remains hard because it involves not just robotic agents moving in static environments but also dynamic human agents and their perceptions of the appropriateness of robot behavior. In contrast, clear, repeatable, and accessible benchmarks have accelerated progress in fields like computer vision, natural language processing and traditional robot navigation by enabling researchers to fairly compare algorithms, revealing limitations of existing solutions and illuminating promising new directions. We believe the same approach can benefit social navigation. In this paper, we pave the road towards common, widely accessible, and repeatable benchmarking criteria to evaluate social robot navigation. Our contributions include (a) a definition of a socially navigating robot as one that respects the principles of safety, comfort, legibility, politeness, social competency, agent understanding, proactivity, and responsiveness to context, (b) guidelines for the use of metrics, development of scenarios, benchmarks, datasets, and simulators to evaluate social navigation, and (c) a design of a social navigation metrics framework to make it easier to compare results from different simulators, robots and datasets.

研究动机与目标

• 基于安全、舒适、可读性、礼貌、社会能力、对其他代理的理解、主动性和情境适当性，给出一个明确、可执行的社交通机器人导航定义。
• 提供评估公平、可重复的指标、情景设计、基准、数据集与仿真器等准则。
• 开发一个社交通导航度量框架，促进跨仿真器和跨数据集的比较。
• 回顾方法学生命周期并提出一个统一的 API，以统一仿真器输出用于评估。

提出的方法

• 提出八条指导原则（P1 安全、P2 舒适、P3 可读性、P4 礼貌、P5 社会能力、P6 理解其他代理、P7 主动性、P8 情境适当性）用于社交通导航。
• 提供一个分类法和结构化指南，将原则与实验设计、指标、情景、基准、数据集和仿真器等组件相连。
• 在统一的评估框架内讨论主观人类评估指标、分析性指标和学习化指标。
• 勾勒社交通导航基准、数据集、仿真器与部署的生命周期，以支持可重复的研究。

Figure 1 : We identify eight broad principles of social robot navigation - including safety, comfort, legibility, politeness, social competency, agent understanding, proactivity, and contextual appropriateness - which motivate specific guidelines for experiments, metrics, scenarios, benchmarks, data

实验结果

研究问题

• RQ1 应该如何定义并将社交通机器人导航分解为可衡量的原则？
• RQ2 需要哪些指标、情景、基准、数据集和仿真器来公平比较社交通导航算法？
• RQ3 如何设计基准和统一的 API，以实现跨仿真器和跨数据集的评估？
• RQ4 评估中情境应如何影响导航原则的权重？
• RQ5 推进社交通导航研究的数据收集、问题发现与实验的实际生命周期应为何？

主要发现

• 提出了八条用于引导社交通导航的原则：安全、舒适、可读性、礼貌、社会能力、对代理的理解、主动性和情境适当性。
• 一种分类法和指南将原则与实验组件（指标、情景、基准、数据集、仿真器）关联起来，以提升可比性。
• 提出一个社交通导航度量框架，便于跨仿真器和跨数据集的比较。
• 描述了社交通导航基准的生命周期，包括数据收集、问题发现、实验室实验和情景开发。
• 本文主张混合现实世界的人机交互研究与基于仿真的消融研究来构建有原则的评估。

Figure 2 : We define a socially navigating robot as one that interacts with humans and other robots in a way that achieves its navigation goals while enabling other agents to achieve theirs. To make this objective achievable, we propose eight principles for social robot navigation: safety, comfort,