[Paper Review] Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning

Meng Zhou, Ziyu Liu+3 more|arXiv (Cornell University)|Jul 6, 2020

Reinforcement Learning in RoboticsComputer Science54 references47 citations

TL;DR

LICA introduces a policy-based CTDE method with a hypernetwork mixing critic and adaptive entropy to implicitly address credit assignment in cooperative MARL, showing strong results on Multi-Agent Particle Environments and StarCraft II micromanagement.

ABSTRACT

We present a multi-agent actor-critic method that aims to implicitly address the credit assignment problem under fully cooperative settings. Our key motivation is that credit assignment among agents may not require an explicit formulation as long as (1) the policy gradients derived from a centralized critic carry sufficient information for the decentralized agents to maximize their joint action value through optimal cooperation and (2) a sustained level of exploration is enforced throughout training. Under the centralized training with decentralized execution (CTDE) paradigm, we achieve the former by formulating the centralized critic as a hypernetwork such that a latent state representation is integrated into the policy gradients through its multiplicative association with the stochastic policies; to achieve the latter, we derive a simple technique called adaptive entropy regularization where magnitudes of the entropy gradients are dynamically rescaled based on the current policy stochasticity to encourage consistent levels of exploration. Our algorithm, referred to as LICA, is evaluated on several benchmarks including the multi-agent particle environments and a set of challenging StarCraft II micromanagement tasks, and we show that LICA significantly outperforms previous methods.

Motivation & Objective

• Motivate learning cooperative policies under shared rewards without explicit credit assignment.
• Develop a centralized critic that provides richer gradient information through a state-conditioned hypernetwork.
• Ensure sustained exploration during training via adaptive entropy regularization.
• Evaluate LICA on challenging cooperative benchmarks and compare to state-of-the-art methods.

Proposed method

• Formulate the centralized critic as a hypernetwork that maps state to a set of weights which mix individual action representations into a joint Q estimate.
• Compute policy gradients through the mixing critic to enable end-to-end optimization of decentralized stochastic policies.
• Train the critic on-policy with a generalized advantage estimation TD(lambda) target and optionally use a target critic for stability.
• Optionally train policies by feeding action distribution parameters to the critic rather than sampled actions to obtain more informative gradients.
• Introduce adaptive entropy regularization that scales entropy gradient magnitudes inversely with current policy entropy to maintain exploration.
• Provide two training regimes: (i) standard action sampling with Gumbel-Softmax for discrete actions, (ii) direct use of action distribution parameters as inputs to the critic.

Experimental results

Research questions

• RQ1 Can implicit credit assignment arise from a state-conditioned mixing critic without explicit baselines or difference rewards?
• RQ2 Does a hypernetwork-based critic improve credit attribution and joint policy quality over standard MLP critics?
• RQ3 How does adaptive entropy regularization affect exploration and convergence in cooperative MARL?
• RQ4 What is the impact of using action distribution parameters as critic inputs on learning efficiency and stability?
• RQ5 Are the proposed methods robust across diverse cooperative tasks and action spaces?

Key findings

• LICA achieves competitive or superior performance compared to prior methods on multi-agent particle environments and StarCraft II micromanagement tasks.
• The mixing critic provides richer state-action gradient information that supports learning cooperative policies beyond monotonic value decompositions.
• Adaptive entropy regularization yields more consistent exploration and faster convergence in complex scenarios.
• Using action distribution parameters as critic inputs can lead to faster and more stable policy convergence in SC2 scenarios.
• Component ablations show the necessity of the mixing critic and the benefits of alternative policy gradient formulations.