[Paper Review] Chat with the Environment: Interactive Multimodal Perception Using Large Language Models

Xufeng Zhao, Mengdi Li+3 more|arXiv (Cornell University)|Mar 14, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Matcha, an interactive multimodal perception framework that uses a Large Language Model (LLM) as a reasoning backbone to guide robots in actively exploring their environment through epistemic actions—such as touching, weighing, or listening—to resolve uncertainty. By integrating modality-specific perception modules that translate sensory data into natural language, Matcha enables the LLM to reason over multimodal feedback, significantly improving object property classification and task success in complex, partially observable environments.

ABSTRACT

Programming robot behavior in a complex world faces challenges on multiple levels, from dextrous low-level skills to high-level planning and reasoning. Recent pre-trained Large Language Models (LLMs) have shown remarkable reasoning ability in few-shot robotic planning. However, it remains challenging to ground LLMs in multimodal sensory input and continuous action output, while enabling a robot to interact with its environment and acquire novel information as its policies unfold. We develop a robot interaction scenario with a partially observable state, which necessitates a robot to decide on a range of epistemic actions in order to sample sensory information among multiple modalities, before being able to execute the task correctly. Matcha (Multimodal environment chatting) agent, an interactive perception framework, is therefore proposed with an LLM as its backbone, whose ability is exploited to instruct epistemic actions and to reason over the resulting multimodal sensations (vision, sound, haptics, proprioception), as well as to plan an entire task execution based on the interactively acquired information. Our study demonstrates that LLMs can provide high-level planning and reasoning skills and control interactive robot behavior in a multimodal environment, while multimodal modules with the context of the environmental state help ground the LLMs and extend their processing ability. The project website can be found at https://matcha-agent.github.io.

Motivation & Objective

• To address the challenge of enabling robots to actively perceive and reason about their environment using multimodal sensory input in partially observable scenarios.
• To overcome the limitations of passive perception by introducing epistemic actions that reduce uncertainty through targeted interaction.
• To ground LLMs in real-world sensory data by translating multimodal inputs into natural language for unified reasoning.
• To demonstrate that LLMs can generalize across novel object properties and instructions by leveraging commonsense knowledge.
• To evaluate the framework's ability to improve classification accuracy and task success through interactive, multimodal feedback loops.

Proposed method

• The Matcha agent uses a Large Language Model (LLM) as its central reasoning engine to generate action commands based on task instructions and environmental context.
• Each modality—vision, sound, haptics, proprioception—is processed by a dedicated, pre-trained perception module that converts raw sensory data into natural language descriptions.
• The LLM uses these natural language feedbacks to reason about object properties, such as material or weight, and to plan the next epistemic action.
• The framework supports iterative 'chatting' with the environment: the robot acts, receives multimodal feedback, and refines its understanding through successive interactions.
• The system is modular and flexible, allowing plug-and-play integration of new perception modules and adaptation to novel objects or tasks.
• The LLM is prompted to perform in-context reasoning, enabling generalization across diverse linguistic formulations of the same task.

Experimental results

Research questions

• RQ1 Can an LLM effectively guide a robot to perform epistemic actions that reduce uncertainty in a partially observable environment?
• RQ2 How does grounding LLM reasoning in multimodal sensory feedback improve object property classification compared to passive perception?
• RQ3 To what extent can LLMs generalize across novel object types and linguistic variations in task instructions?
• RQ4 What role do multimodal perception modules play in enabling the LLM to reason about latent object properties?
• RQ5 How does the interactive, iterative nature of the 'chatting' process enhance task success compared to single-step or passive perception?

Key findings

• The Matcha framework successfully improved object classification accuracy by leveraging interactive perception, particularly in distinguishing materials like glass and ceramic when visual cues were insufficient.
• The integration of multimodal feedback—especially haptics and impact sound—enabled the LLM to correct misclassifications that occurred when relying on a single modality.
• The LLM demonstrated robust generalization across diverse linguistic formulations of the same task, such as shifting from 'pick up the metal block' to 'pick up the block suitable for cracking a nut'.
• Failure cases occurred primarily due to sensing similarity between materials (e.g., glass and ceramic), highlighting the limits of current perception modules in resolving fine-grained epistemic uncertainty.
• The framework showed that strong LLMs can perform complex reasoning and task planning without fine-tuning, relying on in-context learning and commonsense knowledge.
• The modular design of Matcha allows for flexible integration of new perception modules and adaptation to novel environments, enhancing generalizability.