[Paper Review] Learning Nearly Decomposable Value Functions Via Communication Minimization

Tonghan Wang, Jianhao Wang+2 more|arXiv (Cornell University)|Oct 11, 2019

Reinforcement Learning in Robotics31 references44 citations

TL;DR

NDQ learns nearly decomposable Q-functions with communication minimization, enabling agents to act largely independently while selectively exchanging compact messages, reducing communication by over 80% without performance loss in StarCraft II tasks.

ABSTRACT

Reinforcement learning encounters major challenges in multi-agent settings, such as scalability and non-stationarity. Recently, value function factorization learning emerges as a promising way to address these challenges in collaborative multi-agent systems. However, existing methods have been focusing on learning fully decentralized value functions, which are not efficient for tasks requiring communication. To address this limitation, this paper presents a novel framework for learning nearly decomposable Q-functions (NDQ) via communication minimization, with which agents act on their own most of the time but occasionally send messages to other agents in order for effective coordination. This framework hybridizes value function factorization learning and communication learning by introducing two information-theoretic regularizers. These regularizers are maximizing mutual information between agents' action selection and communication messages while minimizing the entropy of messages between agents. We show how to optimize these regularizers in a way that is easily integrated with existing value function factorization methods such as QMIX. Finally, we demonstrate that, on the StarCraft unit micromanagement benchmark, our framework significantly outperforms baseline methods and allows us to cut off more than $80\\%$ of communication without sacrificing the performance. The videos of our experiments are available at https://sites.google.com/view/ndq.

Motivation & Objective

• Address miscoordination and scalability in centralized training with decentralized execution for multi-agent RL.
• Introduce nearly decomposable Q-functions (NDQ) with learnable communication to coordinate when necessary.
• Develop information-theoretic regularizers to maximize expressive communication while minimizing message entropy.
• Integrate NDQ with existing value factorization methods (e.g., QMIX) in a scalable training framework.

Proposed method

• Learn decentralized Q-functions conditioned on local histories and selective messages from other agents.
• Use a stochastic message embedding space where messages are drawn from a distribution parameterized by an encoder, enabling variational optimization.
• Maximize mutual information between other agents’ actions and messages to ensure expressiveness of communication.
• Minimize the entropy (or variability) of messages to ensure succinct communication via a KL-divergence based bound.
• Derive a tractable variational lower bound and integrate with TD loss to train all components end-to-end.
• Enable bit-level cutting-off of messages by thresholding latent message means to drop meaningless communication during execution.

Experimental results

Research questions

• RQ1 Can nearly decomposable Q-functions with minimal communication improve coordination over fully decentralized factorization?
• RQ2 How can information-theoretic regularizers be designed to learn when to communicate and what to send?
• RQ3 What is the impact of communication regularization on performance and robustness when a portion of messages are dropped?
• RQ4 How does NDQ compare to QMIX and TarMAC on challenging MARL benchmarks such as StarCraft II unit micromanagement?

Key findings

• NDQ significantly outperforms baseline factorization methods on StarCraft II unit micromanagement under standard conditions.
• NDQ can reduce communication by more than 80% without sacrificing performance in StarCraft II scenarios.
• Agents learn to coordinate with minimal messaging, sometimes using one or two bits to achieve effective coordination.
• NDQ outperforms attentional communication baselines like QMIX+TarMAC and learns a concise, expressive message embedding.
• When 80% of messages are dropped, NDQ maintains competitive performance, demonstrating robustness to communication reduction.