[Paper Review] MuMMER: Socially Intelligent Human-Robot Interaction in Public Spaces

Mary Ellen Foster, Bart Craenen+28 more|arXiv (Cornell University)|Sep 15, 2019

Social Robot Interaction and HRIPsychology29 references17 citations

TL;DR

MuMMER presents a socially intelligent, autonomous robot system for human-robot interaction in public spaces, integrating audio-visual perception, social signal processing, conversational AI, and human-aware motion planning via ROS. Deployed in a Finnish shopping mall, the system enables natural, multimodal interactions including guided navigation, quizzes, and open-ended dialogue, demonstrating robust real-time performance in real-world conditions with minimal latency through remote processing and modular component integration.

ABSTRACT

In the EU-funded MuMMER project, we have developed a social robot designed to interact naturally and flexibly with users in public spaces such as a shopping mall. We present the latest version of the robot system developed during the project. This system encompasses audio-visual sensing, social signal processing, conversational interaction, perspective taking, geometric reasoning, and motion planning. It successfully combines all these components in an overarching framework using the Robot Operating System (ROS) and has been deployed to a shopping mall in Finland interacting with customers. In this paper, we describe the system components, their interplay, and the resulting robot behaviours and scenarios provided at the shopping mall.

Motivation & Objective

• To develop a socially intelligent robot capable of engaging in natural, flexible interactions with the general public in open, dynamic environments like shopping malls.
• To integrate multiple AI components—audio-visual perception, social signal processing, conversational AI, and motion planning—into a unified, real-time system using ROS.
• To evaluate the system in real-world deployments, including long-term interaction with untrained users in noisy, unpredictable public environments.
• To enable multimodal interaction modes such as chat, quiz, and route guidance with adaptive, context-aware behavior.
• To demonstrate the feasibility of autonomous, socially appropriate robot behavior in real public spaces through long-term deployment and user studies.

Proposed method

• The system uses convolutional pose machines (CPM) and OpenHeadPose for accurate person detection, head pose estimation, and visual tracking in real time.
• A particle filter-based head pose tracker enables re-identification of individuals across frames using color and face features, with OpenFace embeddings for identity matching.
• A multi-task deep neural network processes 4-channel audio input to perform joint speech/non-speech detection and sound source localization, enabling robust speech tracking in noisy environments.
• Speech is fused with visual attention by associating detected speech with the person facing the robot, using head pose as a proxy for visual focus of attention.
• The conversational system uses Alana NLU with Google Translate for Finnish-English translation in the Finnish version, and template-based responses for robustness, while English mode uses full NLU pipelines.
• Task execution is coordinated via a central arbitration module that manages interleaved interaction modes (chat, quiz, guidance) and adapts behavior based on user input and context.

Experimental results

Research questions

• RQ1 How can a robot maintain continuous, socially appropriate interaction with untrained users in a dynamic, noisy public environment like a shopping mall?
• RQ2 What combination of audio-visual perception, social signal processing, and conversational AI enables natural, multimodal human-robot interaction in real time?
• RQ3 How can a robot effectively switch between different interaction modes (e.g., chat, quiz, guidance) while maintaining context and user engagement?
• RQ4 What role does perspective-taking and geometric reasoning play in enabling human-aware navigation and pointing behavior during route guidance?
• RQ5 How can a modular, ROS-based architecture support real-time integration of deep learning components while maintaining low latency in real-world deployment?

Key findings

• The MuMMER system successfully deployed in a shopping mall for three months, demonstrating sustained, real-time interaction with untrained users in a real-world, noisy environment.
• The integration of CPM-based person detection and OpenHeadPose enabled accurate head pose tracking and re-identification of individuals across frames, even when partially occluded.
• The multi-task neural network for audio processing achieved robust sound source localization and speech detection in the noisy mall environment, with performance enhanced by synthetic and semi-automated data collection.
• The system achieved low-latency interaction by offloading heavy computation (ASR, NLU, dialogue management) to remote servers, minimizing delay between user speech and robot response.
• The robot successfully performed complex, interleaved tasks such as guiding users to shops while maintaining conversation, adapting to interruptions, and offering to repeat instructions.
• The use of Google Translate for Finnish-English translation enabled multilingual deployment, though performance limitations led to a simplified template-based response system in the Finnish version, ensuring robustness.