[Paper Review] Improving Generalization in Meta Reinforcement Learning using Neural Objectives

Louis Kirsch, Sjoerd van Steenkiste+1 more|arXiv (Cornell University)|Apr 30, 2020

Reinforcement Learning in Robotics41 references14 citations

TL;DR

MetaGenRL is a meta-reinforcement learning algorithm that distills experiences from diverse agents into a low-complexity neural objective function, enabling strong generalization to unseen environments. It uses off-policy second-order gradients to achieve high sample efficiency and outperforms both standard meta-RL methods and some human-engineered RL algorithms in new environments.

ABSTRACT

Biological evolution has distilled the experiences of many learners into the general learning algorithms of humans. Our novel meta-reinforcement learning algorithm MetaGenRL is inspired by this process. MetaGenRL distills the experiences of many complex agents to meta-learn a low-complexity neural objective function that affects how future individuals will learn. Unlike recent meta-RL algorithms, MetaGenRL can generalize to new environments that are entirely different from those used for meta-training. In some cases, it even outperforms human-engineered RL algorithms. MetaGenRL uses off-policy second-order gradients during meta-training that greatly increase its sample efficiency.

Motivation & Objective

• To develop a meta-RL algorithm that generalizes effectively to environments unseen during meta-training.
• To mimic biological evolution by distilling experiences from multiple agents into a shared, low-complexity neural objective.
• To improve sample efficiency in meta-training through off-policy second-order gradient updates.
• To enable future agents to learn more effectively by learning a generalizable objective function.

Proposed method

• MetaGenRL trains a neural objective function by distilling the experiences of multiple complex agents during meta-training.
• The algorithm uses off-policy data to compute second-order gradients, enhancing sample efficiency during meta-optimization.
• The learned neural objective guides the learning process of future agents, shaping their policy updates.
• The objective function is optimized via meta-gradient descent, allowing it to generalize across diverse environments.
• The method enables transfer to environments that are structurally and dynamically different from meta-training environments.

Experimental results

Research questions

• RQ1 Can a meta-RL algorithm generalize to environments entirely different from those seen during meta-training?
• RQ2 Can a neural objective function learned from diverse agents improve generalization in reinforcement learning?
• RQ3 Does using second-order off-policy gradients in meta-training enhance sample efficiency and performance?
• RQ4 Can a distilled neural objective outperform handcrafted reward functions in new environments?

Key findings

• MetaGenRL generalizes to new environments that are entirely different from those used during meta-training.
• The algorithm outperforms some human-engineered reinforcement learning algorithms in previously unseen environments.
• The use of off-policy second-order gradients significantly improves sample efficiency during meta-training.
• The distilled neural objective function enables effective learning transfer across diverse environments.