[Paper Review] Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments

Maruan Al-Shedivat, Trapit Bansal+4 more|arXiv (Cornell University)|Oct 10, 2017

Domain Adaptation and Few-Shot Learning39 references157 citations

TL;DR

The paper formulates continuous adaptation in nonstationary and competitive RL as gradient-based meta-learning, introduces RoboSumo, and shows meta-learned adaptation outperforms reactive baselines in few-shot settings and in iterated multi-agent games.

ABSTRACT

Ability to continuously learn and adapt from limited experience in nonstationary environments is an important milestone on the path towards general intelligence. In this paper, we cast the problem of continuous adaptation into the learning-to-learn framework. We develop a simple gradient-based meta-learning algorithm suitable for adaptation in dynamically changing and adversarial scenarios. Additionally, we design a new multi-agent competitive environment, RoboSumo, and define iterated adaptation games for testing various aspects of continuous adaptation strategies. We demonstrate that meta-learning enables significantly more efficient adaptation than reactive baselines in the few-shot regime. Our experiments with a population of agents that learn and compete suggest that meta-learners are the fittest.

Motivation & Objective

• Cast continuous adaptation as learning-to-learn for nonstationary environments.
• Develop a gradient-based meta-learning algorithm suitable for rapid adaptation in RL tasks.
• Introduce RoboSumo, a 3D competitive environment, and iterated adaptation games to evaluate adaptation dynamics.
• Demonstrate meta-learning achieves more efficient adaptation than reactive baselines in few-shot scenarios.
• Explore population-level dynamics to assess which adaptation strategies dominate over time.

Proposed method

• Re-derive MAML for multi-task RL from a probabilistic perspective and extend it to dynamic task changes.
• Define adaptation updates that map from initial policy parameters to a task-specific policy using gradient steps.
• Use a meta-loss computed over pairs of consecutive tasks to train both base and adaptation parameters.
• Implement a two-time-scale training and execution procedure with second-order gradient computation for meta-updates.
• Introduce adaptation with importance weighting to handle execution-time data shifts.
• Evaluate with three locomotion nonstationary environments and a RoboSumo multi-agent setting, including iterated adaptation games.

Experimental results

Research questions

• RQ1 Can gradient-based meta-learning enable faster and more robust adaptation to nonstationary or adversarially changing environments than reactive baselines?
• RQ2 How does meta-learning perform in few-shot adaptation scenarios within single-agent locomotion and in competitive multi-agent games?
• RQ3 Do iterated adaptation games promote the emergence of superior adaptation strategies in a population of learning agents?
• RQ4 What is the impact of different policy architectures and adaptation steps on adaptation performance in nonstationary RL tasks?

Key findings

• Meta-learned adaptation substantially outperforms reactive baselines in the few-shot regime across both single-agent locomotion and RoboSumo settings.
• Adaptation via meta-learning enables agents to anticipate environment changes and update policies effectively after only a few experiences.
• In iterated adaptation games, meta-learned strategies often achieve higher win rates and dominate baselines as opponents improve over rounds.
• A diverse population experiment suggests meta-learned adaptation strategies evolve to become the fittest over generations.