[Paper Review] A Multimodal Framework for Human-Multi-Agent Interaction

Shaid Hasan, Breenice Lee+2 more|arXiv (Cornell University)|Mar 24, 2026

Social Robot Interaction and HRI0 citations

TL;DR

The paper presents a multimodal, LLM-driven framework where each humanoid robot is an autonomous cognitive agent with perception, planning, and action modules, coordinated by a central mechanism to enable natural human–multi-agent interaction in shared spaces.

ABSTRACT

Human-robot interaction is increasingly moving toward multi-robot, socially grounded environments. Existing systems struggle to integrate multimodal perception, embodied expression, and coordinated decision-making in a unified framework. This limits natural and scalable interaction in shared physical spaces. We address this gap by introducing a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent with integrated multimodal perception and Large Language Model (LLM)-driven planning grounded in embodiment. At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech and conflicting actions. Implemented on two humanoid robots, our framework enables coherent multi-agent interaction through interaction policies that combine speech, gesture, gaze, and locomotion. Representative interaction runs demonstrate coordinated multimodal reasoning across agents and grounded embodied responses. Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.

Motivation & Objective

• Motivate the need for socially grounded, multi-robot HRI in shared environments.
• Propose a framework where each robot is an autonomous cognitive agent with multimodal perception and embodied action.
• Demonstrate centralized coordination to manage turn-taking and participation among multiple agents.
• Integrate vision–language perception, LLM-driven planning, and action execution within an embodied, modular loop.

Proposed method

• Each robot is a modular closed-loop agent with perception, planning, and action execution.
• Perception uses multimodal input (speech and vision) processed via a vision–language model to produce structured observations.
• Planning employs an LLM conditioned on structured inputs to generate an ordered, parameterized action policy constrained by the robot’s embodied capabilities.
• Action executes a sequence of parameterized primitives (speech, gesture, gaze, locomotion, etc.) and returns status feedback.
• A centralized coordinator evaluates response likelihoods for all agents to regulate turn-taking and participation, ensuring non-overlapping speech and coordinated actions.
• Demonstrations on two humanoid robots illustrate multimodal grounding and coordinated embodiment in interaction scenarios.

Experimental results

Research questions

• RQ1 How can multimodal perception be fused to produce a coherent interaction context for multi-agent HRI?
• RQ2 Can LLM-driven planning generate executable, embodied action policies that respect each agent’s capabilities?
• RQ3 How does centralized coordination affect turn-taking, participation, and grounding in human–multi-agent interactions?
• RQ4 What are the observable effects of embodied actions and latency on perceived coordination and engagement?

Key findings

• The framework enables coherent multi-agent interaction with sequential, non-overlapping speech and grounded embodied responses.
• Each robot’s perception–planning–action loop integrates speech, vision, and embodied behaviors for context-grounded reasoning.
• Centralized coordination prevents conflicting actions and enforces structured turn-taking across agents.
• The system demonstrates distributed reasoning across agents, where each robot reasons from its own perceptual context to generate tailored responses.
• Grounding language into embodied actions is achieved, evidenced by directed addressing and shared interaction context.
• Demonstrations highlight the impact of perception quality and latency on interaction dynamics and perceived coordination.