[Paper Review] Multi-Agent Adversarial Inverse Reinforcement Learning

Lantao Yu, Jiaming Song+1 more|arXiv (Cornell University)|Jul 30, 2019

Anomaly Detection Techniques and Applications45 citations

TL;DR

MA-AIRL is a scalable MaxEnt IRL framework for Markov games that learns reward functions and policies from expert demonstrations using a logistic stochastic best response equilibrium and adversarial training.

ABSTRACT

Reinforcement learning agents are prone to undesired behaviors due to reward mis-specification. Finding a set of reward functions to properly guide agent behaviors is particularly challenging in multi-agent scenarios. Inverse reinforcement learning provides a framework to automatically acquire suitable reward functions from expert demonstrations. Its extension to multi-agent settings, however, is difficult due to the more complex notions of rational behaviors. In this paper, we propose MA-AIRL, a new framework for multi-agent inverse reinforcement learning, which is effective and scalable for Markov games with high-dimensional state-action space and unknown dynamics. We derive our algorithm based on a new solution concept and maximum pseudolikelihood estimation within an adversarial reward learning framework. In the experiments, we demonstrate that MA-AIRL can recover reward functions that are highly correlated with ground truth ones, and significantly outperforms prior methods in terms of policy imitation.

Motivation & Objective

• Motivate the difficulty of reward design in multi-agent systems and the ill-posed nature of IRL in such settings.
• Introduce a new equilibrium concept (logistic stochastic best response equilibrium, LSBRE) suitable for multi-agent IRL.
• Develop MA-AIRL by linking LSBRE to MaxEnt RL and using maximum pseudolikelihood estimation for tractable training.
• Provide a practical adversarial IRL framework that recovers reward functions and enables policy imitation in high-dimensional, unknown-dynamics Markov games.

Proposed method

• Define logistic stochastic best response equilibrium (LSBRE) as a sequence of time-dependent joint policies where each agent best-responds in a stochastic, entropy-regularized manner.
• Show that LSBRE induces a trajectory distribution that can be characterized by an energy-based (MaxEnt) formulation.
• Derive a maximum pseudolikelihood objective that optimizes over agent-wise conditional policies, enabling tractable learning in multi-agent settings.
• Form MA-AIRL as an adversarial learning framework with discriminators parameterized to estimate rewards and adaptive samplers for importance-weighted partition function estimation.
• Use an adaptive sampler q_theta and a reward estimator g_omega with a structured f_{omega,phi} to recover rewards up to potential-based shaping, mitigating reward ambiguity.
• Provide an algorithm (Algorithm 1) that alternates discriminator and generator updates to recover policies and ground-truth-like rewards.

Experimental results

Research questions

• RQ1 Can MA-AIRL efficiently recover expert policies for each agent from demonstrations (policy imitation) in multi-agent Markov games?
• RQ2 Can MA-AIRL accurately recover underlying reward functions that rationalize demonstrations under LSBRE?
• RQ3 How does MA-AIRL compare to prior multi-agent imitation learning methods (e.g., MA-GAIL) in cooperative and competitive tasks?
• RQ4 Does MA-AIRL scale to high-dimensional state-action spaces with unknown dynamics while maintaining reward identifiability?

Key findings

• MA-AIRL recovers reward functions highly correlated with ground truth in experiments.
• MA-AIRL learns policies that significantly outperform state-of-the-art multi-agent imitation learning baselines in mixed cooperative and competitive tasks.
• MA-AIRL extends MaxEnt IRL and adversarial training to Markov games via the LSBRE framework and pseudolikelihood estimation.
• Discriminator outputs align with reward estimation while the adaptive sampler q_theta estimates the expert policy, enabling stable training.
• MA-AIRL demonstrates scalability to high-dimensional state-action spaces and unknown dynamics where previous tabular or simple-structure IRL methods fail.