[Paper Review] RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization

Siqi Shen, Chennan Ma+6 more|arXiv (Cornell University)|Nov 3, 2023

Reinforcement Learning in Robotics4 citations

TL;DR

RiskQ proposes a risk-sensitive multi-agent reinforcement learning algorithm that satisfies the Risk-sensitive Individual-Global-Max (RIGM) principle for risk metrics like Value at Risk (VaR) and distorted risk measures. It models the joint return distribution via weighted quantile mixtures of per-agent return utilities using an attention mechanism, enabling coordinated, decentralized risk-sensitive policies that outperform standard methods in both risk-sensitive and risk-neutral environments.

ABSTRACT

Multi-agent systems are characterized by environmental uncertainty, varying policies of agents, and partial observability, which result in significant risks. In the context of Multi-Agent Reinforcement Learning (MARL), learning coordinated and decentralized policies that are sensitive to risk is challenging. To formulate the coordination requirements in risk-sensitive MARL, we introduce the Risk-sensitive Individual-Global-Max (RIGM) principle as a generalization of the Individual-Global-Max (IGM) and Distributional IGM (DIGM) principles. This principle requires that the collection of risk-sensitive action selections of each agent should be equivalent to the risk-sensitive action selection of the central policy. Current MARL value factorization methods do not satisfy the RIGM principle for common risk metrics such as the Value at Risk (VaR) metric or distorted risk measurements. Therefore, we propose RiskQ to address this limitation, which models the joint return distribution by modeling quantiles of it as weighted quantile mixtures of per-agent return distribution utilities. RiskQ satisfies the RIGM principle for the VaR and distorted risk metrics. We show that RiskQ can obtain promising performance through extensive experiments. The source code of RiskQ is available in https://github.com/xmu-rl-3dv/RiskQ.

Motivation & Objective

• To address the lack of risk-sensitive value factorization in multi-agent reinforcement learning that maintains coordination under risk.
• To formalize coordination requirements in risk-sensitive MARL through the Risk-sensitive Individual-Global-Max (RIGM) principle.
• To design a value factorization method that satisfies RIGM for general risk metrics such as VaR and distorted risk measures.
• To enable decentralized, risk-aware policies that align with centralized risk-sensitive decision-making.
• To empirically validate RiskQ’s performance across risk-sensitive games and StarCraft II tasks.

Proposed method

• Introduces the RIGM principle as a generalization of IGM and DIGM, requiring that joint risk-sensitive actions match individual risk-sensitive greedy actions.
• Models the joint return distribution as a weighted sum of per-agent quantile-based return distribution utilities using an attention mechanism.
• Represents each agent’s risk-sensitive utility via a risk measure $\psi_{\alpha}[Z_i]$, such as VaR or distorted risk, with $\alpha$ as the risk parameter.
• Uses implicit quantile networks (IQN) as a foundation to learn return distribution utilities and enables differentiable quantile regression for policy optimization.
• Employs a quantile mixture mechanism where the quantiles of the joint distribution $Z_{jt}$ are computed as weighted combinations of individual agent quantiles $\theta_i(\omega)$.
• Applies a risk-aware attention mechanism to dynamically assign weights to per-agent quantiles based on risk sensitivity and contribution to joint return distribution.

Experimental results

Research questions

• RQ1 Can a value factorization method be designed to satisfy the RIGM principle for general risk metrics like VaR and distorted risk measures?
• RQ2 How can joint return distributions be modeled in a way that preserves risk-sensitive coordination across agents?
• RQ3 Does the proposed RiskQ method achieve better performance than existing MARL algorithms in risk-sensitive environments?
• RQ4 Can RiskQ maintain strong performance in both risk-neutral and risk-sensitive scenarios?
• RQ5 How does the integration of risk-aware attention and quantile mixture modeling affect policy coordination and robustness?

Key findings

• RiskQ satisfies the RIGM principle for VaR and distorted risk measures, ensuring consistency between decentralized agent actions and centralized risk-sensitive decisions.
• In StarCraft II micromanagement tasks (SMAC), RiskQ achieves superior or competitive performance compared to state-of-the-art MARL baselines, especially under risk-averse settings.
• On risk-sensitive grid-world games, RiskQ demonstrates improved robustness and stability under high-stochasticity and rare-event conditions.
• The ablation study confirms that the attention-based quantile mixture mechanism is critical for maintaining coordination and risk sensitivity.
• Combining RiskQ with exploration strategies like risk-parameter annealing (RiskQ+LQN) yields only marginal gains, suggesting coordinated exploration may require further design.
• Replacing IQN with a more stable risk-aware Bellman operator (Lim and Malik, 2022) leads to performance degradation, indicating that existing distributional RL approximations are sufficient for risk sensitivity.