[Paper Review] Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL

Anusha Nagabandi, Chelsea Finn+1 more|arXiv (Cornell University)|Dec 18, 2018

Domain Adaptation and Few-Shot Learning33 references48 citations

TL;DR

MOLe combines online SGD with a mixture-of-meta-trained-networks using a Chinese restaurant process to adapt continually to non-stationary tasks in model-based RL. Meta-learning provides a favorable prior for rapid online adaptation and task recall.

ABSTRACT

Humans and animals can learn complex predictive models that allow them to accurately and reliably reason about real-world phenomena, and they can adapt such models extremely quickly in the face of unexpected changes. Deep neural network models allow us to represent very complex functions, but lack this capacity for rapid online adaptation. The goal in this paper is to develop a method for continual online learning from an incoming stream of data, using deep neural network models. We formulate an online learning procedure that uses stochastic gradient descent to update model parameters, and an expectation maximization algorithm with a Chinese restaurant process prior to develop and maintain a mixture of models to handle non-stationary task distributions. This allows for all models to be adapted as necessary, with new models instantiated for task changes and old models recalled when previously seen tasks are encountered again. Furthermore, we observe that meta-learning can be used to meta-train a model such that this direct online adaptation with SGD is effective, which is otherwise not the case for large function approximators. In this work, we apply our meta-learning for online learning (MOLe) approach to model-based reinforcement learning, where adapting the predictive model is critical for control; we demonstrate that MOLe outperforms alternative prior methods, and enables effective continuous adaptation in non-stationary task distributions such as varying terrains, motor failures, and unexpected disturbances.

Motivation & Objective

• Motivate rapid, continual online adaptation for deep models in non-stationary environments.
• Develop an online EM-based procedure to maintain and update a mixture of task-specific models.
• Leverage a meta-learned prior to enable effective online adaptation with few gradient steps.
• Handle unknown task boundaries by instantiating new tasks via a CRP prior.

Proposed method

• Formulate online learning as updating theta(T) for each task T with SGD, guided by P(T_t | x_t, y_t).
• Use EM to estimate task responsibilities and update model parameters online.
• Model the task distribution with a Chinese restaurant process to instantiate new tasks as needed.
• Meta-train the prior theta* with MAML to enable rapid adaptation from small data.
• In online M step, update theta_t+1(T_i) using one gradient step scaled by P_t(T_t=T_i|x_t,y_t).
• Apply MOLe to model-based RL by predicting next states from past K transitions and planning with a controller that uses the updated model.

Experimental results

Research questions

• RQ1 Can MOLe autonomously discover task structure in a stream of non-stationary data?
• RQ2 Can MOLe adapt to tasks further outside the training distribution than k-shot methods allow?
• RQ3 Can MOLe recognize and revert to previously seen tasks online?
• RQ4 Does MOLe avoid overfitting to recent tasks and preserve past skills during task switches?
• RQ5 Does MOLe outperform alternative online learning and meta-learning baselines in non-stationary settings?

Key findings

• MOLe enables online instantiation of new tasks and adaptation to out-of-distribution tasks in model-based RL.
• MOLe can recall previously seen tasks and revert to them online.
• MOLe outperforms k-shot meta-RL and baselines using continuous gradient updates or no meta-learning.
• A meta-trained prior via MAML provides effective initialization for continual online adaptation in non-stationary multi-task settings.
• Using a CRP prior allows soft assignment to tasks and natural emergence of specialization without explicit task delineations.