[Paper Review] Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning

Yiqin Yang, Xiaoteng Ma+6 more|arXiv (Cornell University)|Jun 7, 2021

Reinforcement Learning in RoboticsComputer Science55 references18 citations

TL;DR

This paper proposes Implicit Constraint Q-learning (ICQ), a novel offline multi-agent reinforcement learning algorithm that mitigates extrapolation error by restricting Q-value estimation to only state-action pairs present in the offline dataset. By formulating policy learning as a supervised regression under implicit constraints and decomposing joint policies, ICQ achieves state-of-the-art performance on challenging StarCraft II offline multi-agent environments with stable, scalable learning across varying numbers of agents.

ABSTRACT

Learning from datasets without interaction with environments (Offline Learning) is an essential step to apply Reinforcement Learning (RL) algorithms in real-world scenarios. However, compared with the single-agent counterpart, offline multi-agent RL introduces more agents with the larger state and action space, which is more challenging but attracts little attention. We demonstrate current offline RL algorithms are ineffective in multi-agent systems due to the accumulated extrapolation error. In this paper, we propose a novel offline RL algorithm, named Implicit Constraint Q-learning (ICQ), which effectively alleviates the extrapolation error by only trusting the state-action pairs given in the dataset for value estimation. Moreover, we extend ICQ to multi-agent tasks by decomposing the joint-policy under the implicit constraint. Experimental results demonstrate that the extrapolation error is successfully controlled within a reasonable range and insensitive to the number of agents. We further show that ICQ achieves the state-of-the-art performance in the challenging multi-agent offline tasks (StarCraft II). Our code is public online at https://github.com/YiqinYang/ICQ.

Motivation & Objective

• To address the critical challenge of extrapolation error in offline multi-agent reinforcement learning, which accumulates rapidly due to exponential growth in joint action space.
• To develop a method that avoids overgeneralization to unseen state-action pairs by relying solely on observed data for Q-value estimation.
• To extend offline RL principles to multi-agent settings through implicit constraint-based policy decomposition.
• To demonstrate robust and scalable performance across diverse multi-agent offline environments, including complex StarCraft II scenarios.
• To provide the first systematic analysis of extrapolation error in multi-agent offline RL, showing failure modes of existing algorithms like BCQ.

Proposed method

• ICQ formulates Q-value estimation using a SARSA-like update that only considers state-action pairs present in the offline dataset, eliminating reliance on unseen actions.
• It transforms policy learning into a supervised regression problem by optimizing the policy to maximize Q-values over observed state-action pairs, avoiding distributional shift.
• The joint policy is decomposed under implicit constraints, enabling multi-agent cooperation while preserving data fidelity and reducing extrapolation risk.
• A softmax-based normalization is used to approximate the partition function in the policy gradient, ensuring no unseen actions are involved in optimization.
• The method is extended to multi-agent settings via value decomposition, where individual Q-values are combined with shared attention mechanisms to model joint value functions.
• The algorithm uses a Lagrangian relaxation approach to enforce implicit constraints, balancing policy improvement and data consistency.

Experimental results

Research questions

• RQ1 How does extrapolation error scale with the number of agents in offline multi-agent reinforcement learning?
• RQ2 Can offline RL algorithms effectively generalize in multi-agent environments when the joint action space grows exponentially?
• RQ3 To what extent do existing offline RL methods like BCQ fail in multi-agent settings due to unseen action estimation?
• RQ4 Can a method that restricts Q-value estimation to only observed state-action pairs achieve stable and scalable learning in complex multi-agent tasks?
• RQ5 Is it possible to decompose joint policies under implicit constraints to enable effective multi-agent offline learning without interaction?

Key findings

• ICQ successfully controls extrapolation error within a reasonable range regardless of the number of agents, unlike BCQ, which exhibits divergent Q-estimates as agent count increases.
• In the StarCraft II offline benchmark, ICQ achieves state-of-the-art performance across all difficulty levels, including the most challenging 'Super Hard' maps like 27m_vs_30m.
• The single-agent version of ICQ achieves competitive results on the D4RL benchmark, demonstrating its generalization across both single- and multi-agent tasks.
• Ablation studies confirm that the softmax-based partition function approximation introduces minimal bias and maintains performance, validating its use in multi-agent settings.
• ICQ outperforms strong baselines such as BCQ-MA and CQL-MA across all evaluated multi-agent environments, especially in high-complexity scenarios.
• The method shows insensitivity to the number of agents, maintaining stable learning curves and consistent performance even with 30+ agents in the environment.