[Paper Review] Coordinated Multi-Agent Imitation Learning

Hoang Le, Yisong Yue+2 more|arXiv (Cornell University)|Mar 9, 2017

Reinforcement Learning in Robotics25 references60 citations

TL;DR

The paper introduces a semi-supervised framework that jointly learns a latent coordination model and individual policies for multi-agent imitation, using alternating optimization to infer role assignments and improve imitation loss.

ABSTRACT

We study the problem of imitation learning from demonstrations of multiple coordinating agents. One key challenge in this setting is that learning a good model of coordination can be difficult, since coordination is often implicit in the demonstrations and must be inferred as a latent variable. We propose a joint approach that simultaneously learns a latent coordination model along with the individual policies. In particular, our method integrates unsupervised structure learning with conventional imitation learning. We illustrate the power of our approach on a difficult problem of learning multiple policies for fine-grained behavior modeling in team sports, where different players occupy different roles in the coordinated team strategy. We show that having a coordination model to infer the roles of players yields substantially improved imitation loss compared to conventional baselines.

Motivation & Objective

• Motivate imitation learning for multiple coordinating agents where coordination is implicit and roles are unobserved.
• Propose a semi-supervised framework that combines structured latent coordination learning with conventional imitation learning.
• Develop an alternating optimization scheme to train both the latent structure model and the individual policies effectively.
• Demonstrate the approach on synthetic (predator-prey) and real-world-like (professional soccer) multi-agent tasks to show improved imitation performance.

Proposed method

• Formulate coordinated imitation as learning multiple decentralized policies plus a latent coordination model that assigns roles to agents across demonstrations.
• Use a graphical model q to encode the coordination structure and a role-assignment A that re-indexes trajectories to align with learned roles.
• Adopt a reduction-based imitation learning approach for multi-agent policies to enable use of black-box predictors (e.g., deep networks, Random Forests).
• Employ stochastic variational inference to learn q(θ,z) for the coordination structure, with a latent role sequence z modeled as a hidden Markov process.
• Solve the role assignment via a linear assignment problem (Kuhn–Munkres) using a cost matrix derived from the latent model and trajectory likelihoods.
• Train in an alternating fashion (Algorithm 1) between: (i) fixing the structure and learning policies (Algorithm 2), and (ii) updating the latent structure and role assignments (Algorithm 3/Algorithm 4).
• Incorporate entropy regularization on the role assignment to encourage useful diversification of indexings (maximizing H(A|D)).

Experimental results

Research questions

• RQ1 Can a latent coordination model be learned jointly with policies to infer unobserved roles in multi-agent demonstrations?
• RQ2 Does incorporating structured role assignment improve imitation loss compared to unstructured multi-agent imitation learning?
• RQ3 How effective is the alternating optimization framework in addressing non-stationarity and covariate shift in multi-agent imitation?
• RQ4 What is the impact of coordinated role assignments on performance in synthetic (predator-prey) and real-world-like (soccer) domains?

Key findings

• The coordinated approach yields substantially better imitation performance than baselines in both synthetic and soccer domains.
• Role inference through the latent structure model enables more consistent state representations for policy learning and improves coordination.
• The method demonstrates that learning to coordinate via latent roles can scale to large multi-agent settings (e.g., soccer defense with many agents and long trajectories).
• Decentralized policies trained with coordinated role assignments achieve competitive or comparable performance to centralized policies when coordination is learned.
• The approach is the first to apply imitation learning to jointly learn cooperative multi-agent policies at large scale in the presented settings.
• The coordination structure learned (HMM components) reveals dominant modes corresponding to common team formations and role transitions during play.