[Paper Review] Learning Implicit Credit Assignment for Multi-Agent Actor-Critic.

Meng Zhou, Ziyu Liu+3 more|arXiv (Cornell University)|Jul 6, 2020

Reinforcement Learning in Robotics14 references14 citations

TL;DR

This paper proposes LICA, a multi-agent reinforcement learning algorithm that implicitly solves the credit assignment problem by using a hypernetwork-based centralized critic to embed joint policy gradients, while maintaining exploration via adaptive entropy regularization. LICA outperforms prior methods on multi-agent particle environments and StarCraft II micromanagement tasks.

ABSTRACT

We present a new policy-based multi-agent reinforcement learning algorithm that implicitly addresses the credit assignment problem under fully cooperative settings. Our key motivation is that credit assignment may not require an explicit formulation as long as (1) the policy gradients of a trained, centralized critic carry sufficient information for the decentralized agents to maximize the critic estimate through optimal cooperation and (2) a sustained level of agent exploration is enforced throughout training. In this work, we achieve the former by formulating the centralized critic as a hypernetwork such that the latent state representation is now fused into the policy gradients through its multiplicative association with the agent policies, and we show that this is key to learning optimal joint actions that may otherwise require explicit credit assignment. To achieve the latter, we further propose a practical technique called adaptive entropy regularization where magnitudes of the policy gradients from the entropy term are dynamically rescaled to sustain consistent levels of exploration throughout training. Our final algorithm, which we call LICA, is evaluated on several benchmarks including the multi-agent particle environments and a set of challenging StarCraft II micromanagement tasks, and we show that LICA significantly outperforms previous methods.

Motivation & Objective

• To address the credit assignment problem in cooperative multi-agent reinforcement learning without explicit reward decomposition.
• To enable decentralized agents to learn optimal joint actions through policy gradients derived from a centralized critic.
• To maintain consistent exploration levels throughout training using a dynamic entropy regularization technique.
• To improve performance on challenging multi-agent benchmarks such as StarCraft II micromanagement tasks.

Proposed method

• Formulate the centralized critic as a hypernetwork that generates policy gradients by attending to the latent state representation.
• Fuse the latent state representation multiplicatively into the policy gradients, enabling implicit credit assignment through shared value estimation.
• Introduce adaptive entropy regularization that dynamically rescales the entropy term's magnitude to sustain exploration.
• Train the policy networks using policy gradients that are informed by the critic's value estimates and the entropy regularization.
• Use a centralized critic to provide value targets for all agents, enabling joint action optimization without explicit credit assignment.

Experimental results

Research questions

• RQ1 Can implicit credit assignment be achieved through a hypernetwork-based critic that embeds latent state information into policy gradients?
• RQ2 Does maintaining consistent exploration levels through adaptive entropy regularization improve learning stability and performance in multi-agent settings?
• RQ3 Can a policy-based multi-agent algorithm outperform existing methods on complex cooperative tasks without explicit credit assignment?
• RQ4 How does the proposed method scale to challenging environments like StarCraft II micromanagement tasks?

Key findings

• LICA significantly outperforms previous methods on the multi-agent particle environments benchmark.
• The algorithm achieves superior performance on challenging StarCraft II micromanagement tasks, demonstrating robustness in complex cooperative settings.
• Adaptive entropy regularization successfully maintains consistent exploration levels throughout training, improving policy learning stability.
• The hypernetwork-based critic enables implicit credit assignment, allowing agents to learn optimal joint actions without explicit reward decomposition.