[Paper Review] CH-MARL: A Multimodal Benchmark for Cooperative, Heterogeneous Multi-Agent Reinforcement Learning

Vasu Sharma, Prasoon Goyal+4 more|arXiv (Cornell University)|Aug 26, 2022

Reinforcement Learning in Robotics6 citations

TL;DR

CH-MARL introduces a novel multimodal benchmark for cooperative, heterogeneous multi-agent reinforcement learning, featuring a simulated humanoid and drone collaborating in a rich, multi-room home environment using vision and language. The study demonstrates that message passing and multimodal feedback significantly improve task performance, revealing key challenges and opportunities for advancing MARL in complex, real-world-like settings.

ABSTRACT

We propose a multimodal (vision-and-language) benchmark for cooperative and heterogeneous multi-agent learning. We introduce a benchmark multimodal dataset with tasks involving collaboration between multiple simulated heterogeneous robots in a rich multi-room home environment. We provide an integrated learning framework, multimodal implementations of state-of-the-art multi-agent reinforcement learning techniques, and a consistent evaluation protocol. Our experiments investigate the impact of different modalities on multi-agent learning performance. We also introduce a simple message passing method between agents. The results suggest that multimodality introduces unique challenges for cooperative multi-agent learning and there is significant room for advancing multi-agent reinforcement learning methods in such settings.

Motivation & Objective

• To address the lack of standardized, complex environments for evaluating cooperative, heterogeneous multi-agent reinforcement learning (MARL).
• To create a benchmark that supports multimodal inputs (vision and language) and heterogeneous agents with distinct capabilities.
• To investigate the impact of multimodality and communication on MARL performance in collaborative, real-world-like tasks.
• To provide a consistent evaluation protocol and open-source framework for training and testing MARL algorithms.
• To explore the role of natural language feedback and message passing in improving inter-agent collaboration and task efficiency.

Proposed method

• Design a simulated home environment using VirtualHome, featuring a humanoid robot with egocentric vision and physical interaction, and a drone with top-down vision and no physical interaction.
• Develop a procedural language generator to provide natural language feedback that guides agents toward task completion.
• Implement a message passing interface between agents to enable decentralized communication and improve coordination.
• Integrate state-of-the-art MARL algorithms (QMIX, QTRAN, VDN, DQN) with multimodal observations (visual and scene graph) and language feedback.
• Use a consistent evaluation protocol with seen and unseen environments to measure generalization and performance using the PLW (Path Length Weighted) score.
• Train and evaluate models under various settings: with/without language feedback, visual vs. scene graph observations, and with/without message passing.

Experimental results

Research questions

• RQ1 How does multimodal input (vision and language) affect the performance of cooperative MARL algorithms in a heterogeneous multi-agent setup?
• RQ2 To what extent does message passing between agents improve task success and generalization in decentralized MARL settings?
• RQ3 How do different observation modalities (visual vs. scene graph) impact learning efficiency and generalization in unseen environments?
• RQ4 What is the contribution of natural language feedback to the learning process in collaborative task completion?
• RQ5 How do existing MARL algorithms perform on a complex, multimodal, and heterogeneous benchmark compared to behavior cloning baselines?

Key findings

• Message passing-based decentralized MARL models significantly outperform their non-communicative counterparts, with the best-performing model achieving a PLW score of 20.05 on unseen test environments.
• Language feedback provides a 10% relative improvement in success rate over baselines without feedback, demonstrating its critical role in guiding agents.
• Scene graph observations yield substantially better performance (best test-unseen PLW: 20.05) than raw visual observations (best test-unseen PLW: 6.45), indicating challenges in visual feature representation.
• All tested MARL algorithms achieve low PLW scores on unseen environments, highlighting significant room for improvement and the need for new algorithms in multimodal MARL.
• Behavior cloning performs worse than RL-based methods on unseen environments, with a best PLW score of 13.72 compared to 20.05 for the best RL method, indicating generalization advantages of RL.
• The combination of visual input and language feedback yields a success rate of 23.85% on seen test splits, while removing language feedback reduces it to 21.45%, confirming language feedback's measurable benefit.