[Paper Review] Counterfactual Multi-Agent Policy Gradients

Jakob Foerster, Gregory Farquhar+3 more|arXiv (Cornell University)|May 24, 2017

Fuel Cells and Related MaterialsEngineering478 citations

TL;DR

COMA introduces a centralised critic with a per-agent counterfactual baseline to enable effective credit assignment for decentralised policies in cooperative multi-agent RL, demonstrated on decentralised StarCraft micromanagement tasks.

ABSTRACT

Cooperative multi-agent systems can be naturally used to model many real world problems, such as network packet routing and the coordination of autonomous vehicles. There is a great need for new reinforcement learning methods that can efficiently learn decentralised policies for such systems. To this end, we propose a new multi-agent actor-critic method called counterfactual multi-agent (COMA) policy gradients. COMA uses a centralised critic to estimate the Q-function and decentralised actors to optimise the agents' policies. In addition, to address the challenges of multi-agent credit assignment, it uses a counterfactual baseline that marginalises out a single agent's action, while keeping the other agents' actions fixed. COMA also uses a critic representation that allows the counterfactual baseline to be computed efficiently in a single forward pass. We evaluate COMA in the testbed of StarCraft unit micromanagement, using a decentralised variant with significant partial observability. COMA significantly improves average performance over other multi-agent actor-critic methods in this setting, and the best performing agents are competitive with state-of-the-art centralised controllers that get access to the full state.

Motivation & Objective

• Motivate the need for decentralised policies in cooperative multi-agent RL and address credit assignment under global rewards.
• Propose COMA: a multi-agent actor-critic method with a centralised critic and a counterfactual baseline.
• Show how a specialised critic representation enables efficient computation of the counterfactual baseline in one forward pass.
• Empirically evaluate COMA on decentralised StarCraft micromanagement tasks with partial observability, comparing against baselines.

Proposed method

• Use a centralised critic during training that conditions on the joint action and state information.
• Define a per-agent counterfactual baseline that marginalises out the agent’s action while keeping others fixed to form an advantage A^a(s,u) = Q(s,u) - sum_{u^a'} pi^a(u^a'|tau^a) Q(s,(u^{-a},u^a')).
• Compute Q-values for all agents efficiently with a critic that outputs Q-values for each of an agent’s actions given others’ actions, enabling a single forward pass.
• Ground the approach in policy gradient with g = E_pi[ sum_a ∇_theta log pi^a(u^a|tau^a) A^a(s,u) ].
• Adapt TD(lambda) critics for on-policy learning with a target network for Q or V estimation.
• Empirically evaluate COMA on StarCraft unit micromanagement with partial observability and compare to IAC baselines and centralised controls.

Experimental results

Research questions

• RQ1 Does a centralised critic with a counterfactual baseline improve credit assignment for decentralised agents under shared global rewards?
• RQ2 Can COMA outperform standard multi-agent actor-critic baselines (IAC variants) and remain competitive with centralised controllers in partially observable StarCraft micromanagement tasks?
• RQ3 Is the proposed critic representation for efficient counterfactual baseline evaluation effective in practice?
• RQ4 How does COMA perform across different multi-agent team sizes and map difficulties under limited field of view?

Key findings

• COMA outperforms IAC baselines across all StarCraft scenarios in win rate.
• CENTRAL-QV baselines are outperformed by COMA, indicating the importance of the counterfactual baseline.
• COMA learns faster and with more stability than central V baselines, due to its shaped training signal.
• Best COMA agents achieve competitive performance with state-of-the-art centralised controllers given full state access and macro-actions.
• Ablations show centralised critic plus counterfactual baseline are crucial for final performance and learning efficiency.