[Paper Review] Learning with AMIGo: Adversarially Motivated Intrinsic Goals

Andres Campero, Roberta Răileanu+4 more|arXiv (Cornell University)|Jun 22, 2020

Adversarial Robustness in Machine Learning57 references48 citations

TL;DR

AMIGo trains a goal-generating teacher to propose increasingly challenging intrinsic goals for a student policy, creating a self-driven curriculum that enables learning in sparse-reward environments and surpasses state-of-the-art intrinsic motivation baselines on challenging MiniGrid tasks.

ABSTRACT

A key challenge for reinforcement learning (RL) consists of learning in environments with sparse extrinsic rewards. In contrast to current RL methods, humans are able to learn new skills with little or no reward by using various forms of intrinsic motivation. We propose AMIGo, a novel agent incorporating -- as form of meta-learning -- a goal-generating teacher that proposes Adversarially Motivated Intrinsic Goals to train a goal-conditioned "student" policy in the absence of (or alongside) environment reward. Specifically, through a simple but effective "constructively adversarial" objective, the teacher learns to propose increasingly challenging -- yet achievable -- goals that allow the student to learn general skills for acting in a new environment, independent of the task to be solved. We show that our method generates a natural curriculum of self-proposed goals which ultimately allows the agent to solve challenging procedurally-generated tasks where other forms of intrinsic motivation and state-of-the-art RL methods fail.

Motivation & Objective

• Motivate a learning framework capable of solving sparse-reward RL tasks via intrinsic motivation.
• Propose a goal-generating teacher that creates challenging yet achievable intrinsic goals for a student policy.
• Demonstrate that automatic curriculum generation improves sample efficiency and task generalization in procedurally generated environments.

Proposed method

• Introduce AMIGo, consisting of a goal-generating teacher G and a goal-conditioned student π.
• Teacher G proposes a goal g at episode start or when reached; student receives intrinsic reward r^g based on achieving g, plus environment reward r^e.
• Teacher reward r^T rewards goals that are hard but achievable by the student, using a threshold t* to control goal difficulty.
• Student optimizes discounted return R_t with rewards r_t = r^g_t + r^e_t.
• The environments are MiniGrid tasks with procedurally generated layouts; goals are defined as changes to tile observations via coordinates (x,y).
• An auxiliary loss set (diversity, episode boundary awareness, extrinsic goal alignment) is explored but not essential to AMIGo.

Experimental results

Research questions

• RQ1 Can a goal-generating teacher learn to create a curriculum of intrinsically motivated goals that progressively increases task difficulty for a student policy?
• RQ2 Does AMIGo outperform state-of-the-art intrinsic motivation baselines on challenging, procedurally generated MiniGrid environments?
• RQ3 How does the ablation of auxiliary teacher losses affect learning and performance?
• RQ4 Is the framework architecture-agnostic and adaptable to different RL models and goal modalities?

Key findings

• AMIGo achieves state-of-the-art results on several hard MiniGrid tasks and solves some environments where other methods fail.
• AMIGo outperforms baselines such as IMPALA without intrinsic motivation, Count, ICM, RND, RIDE, and ASP on hard environments.
• In medium environments, AMIGo performs comparably to other strong intrinsic motivation methods.
• AMIGo demonstrates a natural curriculum where the teacher increases goal difficulty as the student improves.
• Qualitative analysis shows cooperative and adversarial dynamics between teacher and student as learning progresses.