[Paper Review] Role Playing Learning for Socially Concomitant Mobile Robot Navigation

Mingming Li, Rui Jiang+2 more|arXiv (Cornell University)|May 29, 2017

Reinforcement Learning in Robotics46 references3 citations

TL;DR

This paper proposes Role Playing Learning (RPL), a reinforcement learning framework that enables mobile robots to navigate socially concomitantly with a human companion in dynamic environments. By simulating interactions using real pedestrian trajectories and optimizing a neural network policy via PO-TRPO, the robot learns to maintain proximity to its companion while cooperatively avoiding obstacles, achieving human-like navigation behavior with 1.05m average companion distance and 0.49m minimal pedestrian distance in real-world tests.

ABSTRACT

In this paper, we present the Role Playing Learning (RPL) scheme for a mobile robot to navigate socially with its human companion in populated environments. Neural networks (NN) are constructed to parameterize a stochastic policy that directly maps sensory data collected by the robot to its velocity outputs, while respecting a set of social norms. An efficient simulative learning environment is built with maps and pedestrians trajectories collected from a number of real-world crowd data sets. In each learning iteration, a robot equipped with the NN policy is created virtually in the learning environment to play itself as a companied pedestrian and navigate towards a goal in a socially concomitant manner. Thus, we call this process Role Playing Learning, which is formulated under a reinforcement learning (RL) framework. The NN policy is optimized end-to-end using Trust Region Policy Optimization (TRPO), with consideration of the imperfectness of robot's sensor measurements. Simulative and experimental results are provided to demonstrate the efficacy and superiority of our method.

Motivation & Objective

• Address the challenge of socially concomitant navigation (SCN), where a robot must navigate with a human companion while avoiding collisions with others.
• Overcome limitations of traditional methods that treat pedestrians as static obstacles or assume shared decision-making models.
• Develop a generalizable, sensor-robust navigation policy that does not require global maps, known goals, or domain-specific features.
• Enable efficient and safe policy learning through a simulative environment mirroring real-world pedestrian trajectories.
• Ensure the robot maintains a natural, comfortable distance (1.05m avg.) from its companion while preserving safety from other pedestrians (0.49m min. distance).

Proposed method

• Formulate SCN as a Partially Observable Markov Decision Process (POMDP) to account for sensor noise and incomplete observations.
• Construct a simulative learning environment using real-world crowd trajectory datasets to mirror pedestrian behaviors and corridor dynamics.
• Train a neural network policy using Partially Observable Trust Region Policy Optimization (PO-TRPO), which handles observation uncertainty and enables on-policy learning.
• Implement Role Playing Learning (RPL), where the robot virtually 'plays itself' as a companion in each training iteration, simulating joint navigation with a randomly selected pedestrian.
• Optimize the policy end-to-end using sparse rewards based on proximity to goal, collision avoidance, and companion distance metrics.
• Incorporate sensor noise and limited field-of-view constraints directly into the observation space to reflect real robot limitations.

Experimental results

Research questions

• RQ1 Can a robot learn to navigate cooperatively with a human companion in crowded, dynamic environments without prior knowledge of the companion’s goal?
• RQ2 How can a robot maintain a socially acceptable distance from its companion (1.05m avg.) while avoiding collisions with other pedestrians?
• RQ3 To what extent can a policy trained in a simulated environment with real pedestrian trajectories generalize to unseen real-world scenarios?
• RQ4 Can the robot achieve human-like cooperative behavior in joint collision avoidance without assuming shared decision-making models?
• RQ5 How does sensor noise and partial observability affect the robustness and performance of socially concomitant navigation policies?

Key findings

• The robot achieved a minimal average distance of 0.49m from oncoming pedestrians during joint collision avoidance, demonstrating safe and effective obstacle navigation.
• The average companion distance was 1.05m, closely matching the human control baseline of 1.0m, indicating strong companionship behavior and natural gait alignment.
• The robot successfully avoided the 'freezing robot problem' by proactively yielding space, such as moving toward the wall to allow others to pass.
• The policy generalized well to real-world scenarios, with performance comparable to human-driven navigation in both social navigation and SCN tasks.
• The RPL framework enabled end-to-end training without requiring global maps, known goals, or manually engineered features, enhancing generalizability.
• PO-TRPO effectively optimized the policy under partial observability, showing robustness to sensor noise and limited field-of-view constraints.