[Paper Review] Mix & Match Agent Curricula for Reinforcement Learning

Wojciech Marian Czarnecki, Siddhant M. Jayakumar+6 more|arXiv (Cornell University)|Jun 5, 2018

Reinforcement Learning in Robotics20 references29 citations

TL;DR

Mix & Match (M&M) is a curriculum learning framework for reinforcement learning that trains a complex target agent by progressively distilling knowledge from simpler, easier-to-train agent variants, using a dynamic mixture of policies and KL divergence-based distillation. It enables faster, more stable training of agents with large action spaces or complex architectures, achieving superior performance in 3D environments compared to standard methods.

ABSTRACT

We introduce Mix&Match (M&M) - a training framework designed to facilitate rapid and effective learning in RL agents, especially those that would be too slow or too challenging to train otherwise. The key innovation is a procedure that allows us to automatically form a curriculum over agents. Through such a curriculum we can progressively train more complex agents by, effectively, bootstrapping from solutions found by simpler agents. In contradistinction to typical curriculum learning approaches, we do not gradually modify the tasks or environments presented, but instead use a process to gradually alter how the policy is represented internally. We show the broad applicability of our method by demonstrating significant performance gains in three different experimental setups: (1) We train an agent able to control more than 700 actions in a challenging 3D first-person task; using our method to progress through an action-space curriculum we achieve both faster training and better final performance than one obtains using traditional methods. (2) We further show that M&M can be used successfully to progress through a curriculum of architectural variants defining an agents internal state. (3) Finally, we illustrate how a variant of our method can be used to improve agent performance in a multitask setting.

Motivation & Objective

• To address the challenge of training complex reinforcement learning agents that are too slow or unstable to train from scratch.
• To overcome limitations of traditional curriculum learning, which requires task-level modifications and is inapplicable when environment design is fixed.
• To develop a method that enables knowledge transfer from simpler, easier-to-train agents to a final, complex target agent through architectural curriculum.
• To improve data efficiency and final performance in challenging 3D environments with large action spaces or complex agent architectures.
• To provide a generalizable framework applicable to diverse RL settings without requiring modifications to the environment or task specification.

Proposed method

• The framework uses a mixture policy πmm = Σ αiπi, where αi are time-varying mixing coefficients that shift from simpler to more complex agents over training.
• Knowledge distillation is applied via KL divergence loss: L = (1−α)DKL(π1∥π2), encouraging the complex agent to match the behavior of simpler agents early in training.
• The mixing coefficient α is gradually increased from α1=1 (only simple agent) to αK=1 (only target agent), enabling a curriculum over agent architectures.
• The method supports both action-space curricula (varying action space size) and architectural curricula (varying internal state representations).
• Distillation is applied online during training, with the final policy using only the target agent's policy for experience collection.
• The approach is compatible with standard RL algorithms and can be combined with task-based curricula when environment modifications are possible.

Experimental results

Research questions

• RQ1 Can a curriculum over agent architectures, rather than tasks, improve training efficiency and final performance in deep reinforcement learning?
• RQ2 How can knowledge be effectively transferred from simpler, easier-to-train agents to a complex target agent without modifying the environment?
• RQ3 Does progressive training via policy mixture and distillation lead to faster convergence and better final performance in large-action-space 3D environments?
• RQ4 Can the M&M framework be applied to train agents with complex internal state representations, such as recurrent or memory-augmented networks?
• RQ5 How does the dynamic mixing of policies affect learning stability and data efficiency compared to standard training?

Key findings

• In a 3D first-person navigation task with over 700 actions, M&M achieved faster training and better final performance than standard methods.
• The method enabled successful training of a complex agent with a large action space that was previously too difficult to train directly.
• M&M improved performance in a multitask setting by enabling effective learning through a curriculum of architectural variants.
• The use of distillation with a time-varying mixture policy led to stable knowledge transfer and reduced training instability.
• The framework demonstrated broad applicability across different types of agent complexity, including action space size and internal architecture.
• Empirical results showed that the proposed distillation loss with time-varying α effectively guided the complex agent toward the behavior of simpler predecessors.