[Paper Review] Actor-Attention-Critic for Multi-Agent Reinforcement Learning

Shariq Iqbal, Fei Sha|arXiv (Cornell University)|Oct 5, 2018

Reinforcement Learning in Robotics30 references289 citations

TL;DR

MAAC introduces a centralized critic with an attention mechanism to selectively attend to other agents, enabling scalable, decentralized policies across cooperative, competitive, and mixed MARL settings.

ABSTRACT

Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings. We present an actor-critic algorithm that trains decentralized policies in multi-agent settings, using centrally computed critics that share an attention mechanism which selects relevant information for each agent at every timestep. This attention mechanism enables more effective and scalable learning in complex multi-agent environments, when compared to recent approaches. Our approach is applicable not only to cooperative settings with shared rewards, but also individualized reward settings, including adversarial settings, as well as settings that do not provide global states, and it makes no assumptions about the action spaces of the agents. As such, it is flexible enough to be applied to most multi-agent learning problems.

Motivation & Objective

• Address non-stationarity and scalability in multi-agent RL by learning centralized critics that attend to only relevant other agents at each time step.
• Allow heterogeneous reward structures and action spaces across agents while maintaining decentralized execution.
• Improve credit assignment through an attention-based multi-agent baseline and entropy-regularized learning.
• Demonstrate scalability and adaptability in cooperative, competitive, and mixed environments without requiring global states.

Proposed method

• Use centralized critics for each agent with an attention mechanism to weigh information from other agents.
• Represent Q_i as Q_i^ψ(o,a) = f_i(g_i(o_i,a_i), x_i) where x_i is a weighted sum of other agents’ encoded information.
• Compute attention weights α_j via a bilinear query-key mechanism with shared parameters across agents, potentially using multiple heads.
• Train critics jointly using a TD-style loss with a multi-agent baseline to compute A_i(o,a) = Q_i^ψ(o,a) − b(o,a_{ eq i}).
• Update individual policies θ_i using a policy gradient that includes entropy regularization and the multi-agent baseline in the advantage term.
• Optionally extend to exact expectation of Q for discrete actions by outputting Q across all possible actions for each agent when computing the baseline.

Experimental results

Research questions

• RQ1 Can a centralized attention-based critic improve learning stability and scalability in multi-agent settings compared to prior centralized training methods?
• RQ2 Does dynamic attention to relevant agents improve credit assignment and performance across cooperative, competitive, and mixed MARL environments?
• RQ3 How does MAAC perform when agents have different reward structures and action spaces, and when global states are unavailable?
• RQ4 Does MAAC scale better with the number of agents than concatenation-based critics in complex interaction scenarios?

Key findings

• MAAC achieves competitive performance in cooperative and mixed environments, and shows improved scalability over baselines as the number of agents grows.
• The attention mechanism enables agents to focus on relevant others (e.g., Rover-Tower) and can be visualized to attend to the paired agent, without explicit supervision.
• MAAC with learned attention scales better than MADDPG+SAC in the Cooperative Treasure Collection as agent count increases.
• Uniform attention can be competitive in some tasks, but dynamic attention provides advantages in scenarios with shifting relevant partners and subgroup interactions (e.g., Rover-Tower).
• Empirical results indicate MAAC maintains performance while the number of agents increases, unlike some baselines whose performance deteriorates with scale.