[Paper Review] I can attend a meeting too! Towards a human-like telepresence avatar robot to attend meeting on your behalf

Hrishav Bakul Barua, Chayan Sarkar+3 more|arXiv (Cornell University)|Jun 28, 2020

Speech and Audio Processing13 references4 citations

TL;DR

This paper presents a human-like telepresence robot that autonomously attends meetings on behalf of a remote user by using audio-visual perception to localize speakers and shift attention naturally. It combines time difference of arrival (TDOA) for sound source localization with face and lip movement detection, enabling real-time, context-aware robot rotation that enhances user belongingness and satisfaction in group meetings.

ABSTRACT

Telepresence robots are used in various forms in various use-cases that helps to avoid physical human presence at the scene of action. In this work, we focus on a telepresence robot that can be used to attend a meeting remotely with a group of people. Unlike a one-to-one meeting, participants in a group meeting can be located at a different part of the room, especially in an informal setup. As a result, all of them may not be at the viewing angle of the robot, a.k.a. the remote participant. In such a case, to provide a better meeting experience, the robot should localize the speaker and bring the speaker at the center of the viewing angle. Though sound source localization can easily be done using a microphone-array, bringing the speaker or set of speakers at the viewing angle is not a trivial task. First of all, the robot should react only to a human voice, but not to the random noises. Secondly, if there are multiple speakers, to whom the robot should face or should it rotate continuously with every new speaker? Lastly, most robotic platforms are resource-constrained and to achieve a real-time response, i.e., avoiding network delay, all the algorithms should be implemented within the robot itself. This article presents a study and implementation of an attention shifting scheme in a telepresence meeting scenario which best suits the needs and expectations of the collocated and remote attendees. We define a policy to decide when a robot should rotate and how much based on real-time speaker localization. Using user satisfaction study, we show the efficacy and usability of our system in the meeting scenario. Moreover, our system can be easily adapted to other scenarios where multiple people are located.

Motivation & Objective

• To enable a telepresence robot to attend group meetings on behalf of a remote user with minimal manual intervention.
• To address the challenge of dynamic attention shifting in multi-speaker meeting environments where the robot must identify and focus on the current speaker.
• To improve the sense of belongingness and social presence for both remote and collocated attendees through human-like attention behavior.
• To design a real-time, on-device system suitable for resource-constrained robotic platforms like TurtleBot2.
• To evaluate the system’s usability and user satisfaction in realistic meeting scenarios with varying numbers of participants.

Proposed method

• Utilizes time difference of arrival (TDOA) with a Raspberry Pi-based microphone array for real-time sound source localization (SSL).
• Integrates face detection and lip movement detection to verify and refine speaker localization for visual attention.
• Employs a robot state representation model to manage attention shifting between speakers in multi-person scenarios.
• Applies decision rules to determine when and how much to rotate the robot based on speaker activity and group dynamics.
• Implements the entire system on a TurtleBot2 platform with onboard processing to minimize network delay and ensure real-time responsiveness.
• Uses an adjustable camera mount to optimize face detection in seated meeting configurations.

Experimental results

Research questions

• RQ1 How can a telepresence robot autonomously detect and focus on the current speaker in a dynamic group meeting setting?
• RQ2 What attention-shifting strategy best balances responsiveness, accuracy, and natural human-like behavior in multi-speaker meetings?
• RQ3 To what extent does the robot’s autonomous behavior enhance the sense of belongingness and satisfaction for both remote and collocated attendees?
• RQ4 How does the system perform in terms of attention accuracy and unnecessary turning in real-world meeting scenarios?
• RQ5 Can the proposed audio-visual perception and state model be effectively deployed on a resource-constrained robot platform in real time?

Key findings

• The robot achieved high user satisfaction, with remote and collocated attendees rating the system’s sense of belongingness at 7 out of 10 on average.
• The system successfully minimized unnecessary turns and missed necessary turns, with evaluation metrics showing low error rates in attention shifting.
• Speaker detection accuracy was high, with the robot correctly identifying the speaker in most cases, especially in one-on-one and small group settings.
• The attention accuracy in groups of two or more people was consistently high, with the robot correctly focusing on the active speaker in most interactions.
• The audio-visual perception system enabled the robot to behave in a human-like manner—shifting focus naturally between speakers and maintaining eye contact—without requiring remote user control.
• User feedback confirmed that the robot enhanced the feeling of personal presence and social inclusion for remote attendees, particularly in multi-person meetings.