[Paper Review] Learning Policy Representations in Multiagent Systems

Aditya Grover, Maruan Al-Shedivat+3 more|arXiv (Cornell University)|Jun 17, 2018

Reinforcement Learning in Robotics19 references43 citations

TL;DR

Proposes an unsupervised encoder-decoder framework to learn continuous policy embeddings from few interactions, enabling downstream tasks like clustering, outcome prediction, and policy optimization in both competitive and cooperative MAS.

ABSTRACT

Modeling agent behavior is central to understanding the emergence of complex phenomena in multiagent systems. Prior work in agent modeling has largely been task-specific and driven by hand-engineering domain-specific prior knowledge. We propose a general learning framework for modeling agent behavior in any multiagent system using only a handful of interaction data. Our framework casts agent modeling as a representation learning problem. Consequently, we construct a novel objective inspired by imitation learning and agent identification and design an algorithm for unsupervised learning of representations of agent policies. We demonstrate empirically the utility of the proposed framework in (i) a challenging high-dimensional competitive environment for continuous control and (ii) a cooperative environment for communication, on supervised predictive tasks, unsupervised clustering, and policy optimization using deep reinforcement learning.

Motivation & Objective

• Motivate learning generalizable representations of agent policies from limited interaction data in multiagent systems.
• Develop an encoder-decoder framework that learns embeddings of agent interaction episodes.
• Combine generative (imitation learning) and discriminative (agent identification) objectives to learn useful policy representations.
• Demonstrate that embeddings improve downstream tasks including clustering, outcome prediction, and policy optimization in both competitive and cooperative MAS.
• Provide a generalization framework for MAS representations across unseen agents and interactions using agent-interaction graphs.

Proposed method

• Define a representation function f_theta: E -> R^d that maps interaction episodes to embeddings.
• Train a conditional policy network pi_{phi,theta} that uses embeddings to select actions, optimizing imitation-like cross-entropy loss.
• Introduce a triplet-based discriminative loss on embeddings to encourage agent-specific clustering of embeddings.
• Combine imitation-based generative loss and triplet-based discriminative loss into a hybrid objective: Im_loss + lambda * Id_loss (Equation 3).
• Use two distinct episodes per update (e1, e2) to learn embeddings that condition one episode’s policy on another’s embedding.
• Run experiments in RoboSumo (competitive) and ParticleWorld (cooperative communication) to evaluate embedding usefulness for clustering, outcome prediction, and policy optimization.
• Represent generalization via agent-interaction graphs to study weak/strong/generalization across unseen agents, interactions, and tasks.

Experimental results

Research questions

• RQ1 Can unsupervised embeddings learned from limited interactions generalize to unseen agents and interactions in MAS?
• RQ2 Do generative (imitation) and discriminative (agent identification) objectives complement to create more useful policy representations?
• RQ3 Are learned embeddings effective for downstream tasks such as clustering policies, predicting outcomes, and improving policy learning in new opponents or listeners?
• RQ4 Can embeddings conditioned policies adapt faster and generalize better in both competitive and cooperative MAS settings?

Key findings

• Emb-Hyb (hybrid Im+Id) generally yields better clustering signal (lower intra-inter cluster ratio) and competitive outcome prediction than Im or Id alone in RoboSumo.
• Embeddings enable improved policy optimization, with Emb-Hyb providing the best generalization to unseen opponents in RoboSumo during PPO-based training.
• In ParticleWorld, embeddings facilitate better listener-speaker coordination and improve average rewards when used to condition speaker policies, especially with offline embeddings.
• Embeddings learned online can outperform baseline but offline embeddings tend to offer stronger generalization in policy optimization tasks.
• The framework demonstrates strong qualitative separation of agent policies in embedding space, with embeddings revealing agent-specific structure across episodes.