[Paper Review] Multiagent Soft Q-Learning

Ermo Wei, Drew Wicke+2 more|arXiv (Cornell University)|Apr 25, 2018

Reinforcement Learning in RoboticsComputer Science22 references45 citations

TL;DR

The paper introduces Multiagent Soft Q-Learning, an off-policy, centralized-training/decentralized-execution method that uses soft Q-learning and deep energy-based policies to mitigate relative overgeneralization in cooperative continuous games and to improve joint-action coordination.

ABSTRACT

Policy gradient methods are often applied to reinforcement learning in continuous multiagent games. These methods perform local search in the joint-action space, and as we show, they are susceptable to a game-theoretic pathology known as relative overgeneralization. To resolve this issue, we propose Multiagent Soft Q-learning, which can be seen as the analogue of applying Q-learning to continuous controls. We compare our method to MADDPG, a state-of-the-art approach, and show that our method achieves better coordination in multiagent cooperative tasks, converging to better local optima in the joint action space.

Motivation & Objective

• Motivate the challenge of coordination in cooperative multiagent reinforcement learning with continuous actions and the relative overgeneralization pathology.
• Propose a centralized-training, decentralized-execution framework to improve coordination in continuous games.
• Develop a Multiagent Soft Q-Learning method that integrates soft Q-learning with deep energy-based policies to avoid overgeneralization and improve local optima in joint action space.

Proposed method

• Formulate the problem within cooperative stochastic games and analyze the relative overgeneralization pathology for policy gradient methods.
• Adopt Soft Q-Learning with deep energy-based policies to enable multimodal action distributions and better exploration of the joint action space.
• Use a centralized critic shared across agents and individual agents’ policies, with annealing of the entropy temperature alpha to drive exploration then convergence.
• Explain and implement a multiagent extension where agents learn the joint-action policy via the central Q-function and sample from a softmax-like energy-based policy using SVGD for approximation.
• Provide Algorithm 1 that outlines centralized critic updates and per-agent joint-action policy updates.
• Perform empirical evaluation in a two-agent, single-state continuous game to compare against MADDPG and demonstrate improved convergence to better equilibria.

Experimental results

Research questions

• RQ1 Can centralized training with a soft Q-learning based approach overcome relative overgeneralization in cooperative continuous games?
• RQ2 Does employing deep energy-based policies with entropy regularization improve multimodal exploration and coordination in the joint action space?
• RQ3 How does Multiagent Soft Q-Learning compare to MADDPG in terms of convergence to better local optima in cooperative tasks?
• RQ4 What is the effect of annealing the temperature parameter alpha on discovering and locking onto better joint actions?

Key findings

• Multiagent Soft Q-Learning converges to better equilibria more often than MADDPG in the tested continuous coordination task (72% vs MADDPG never converged to the better equilibrium).
• Annealing alpha helps the method transition from global joint-action exploration to exploiting better joint actions, leading to improved coordination.
• The approach leverages centralized training to access the joint-action Q-function while maintaining decentralized execution, reducing co-adaptation.
• The method inherits advantages of Soft Q-Learning, enabling multimodal action distributions and better exploration of the joint action space in continuous games.
• The empirical evaluation uses a Max of Two Quadratics game to illustrate improved coordination and local optima in joint action space.