[Paper Review] CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning

Cédric Colas, Pierre Fournier+3 more|arXiv (Cornell University)|Oct 15, 2018

Reinforcement Learning in Robotics35 references42 citations

TL;DR

CURIOUS presents a modular, goal-conditioned RL approach (m-uvfa) combined with intrinsic learning-progress based curriculum to autonomously select and learn diverse modular goals within a single policy, enabling developmental self-organization and robustness to distractions and perturbations.

ABSTRACT

In open-ended environments, autonomous learning agents must set their own goals and build their own curriculum through an intrinsically motivated exploration. They may consider a large diversity of goals, aiming to discover what is controllable in their environments, and what is not. Because some goals might prove easy and some impossible, agents must actively select which goal to practice at any moment, to maximize their overall mastery on the set of learnable goals. This paper proposes CURIOUS, an algorithm that leverages 1) a modular Universal Value Function Approximator with hindsight learning to achieve a diversity of goals of different kinds within a unique policy and 2) an automated curriculum learning mechanism that biases the attention of the agent towards goals maximizing the absolute learning progress. Agents focus sequentially on goals of increasing complexity, and focus back on goals that are being forgotten. Experiments conducted in a new modular-goal robotic environment show the resulting developmental self-organization of a learning curriculum, and demonstrate properties of robustness to distracting goals, forgetting and changes in body properties.

Motivation & Objective

• Enable learning of a continuous set of diverse modular goals within a single policy using modular goal representations.
• Introduce an intrinsic motivation mechanism based on absolute learning progress to guide module and goal replay.
• Develop a monolithic modular policy (m-uvfa) that facilitates cross-module and cross-goal knowledge transfer.
• Create and evaluate a modular goal environment to study self-organized curricula and robustness.

Proposed method

• Extend Universal Value Function Approximators (uvfa) with modular goal inputs to form Modular-uvfa (m-uvfa).
• Use a one-hot module descriptor and goal masking to handle multiple goal modules within a single network.
• Train with off-policy methods (DDPG) and hindsight-like goal substitution for cross-goal learning.
• Implement intrinsic motivation by estimating competence and learning progress per module and selecting modules via a proportional probability matching policy with epsilon-greedy exploration.
• Employ cross-module and cross-goal replay by substituting module descriptors and goals in experiences, augmented with internal rewards based on imagined module-goal constraints.

Experimental results

Research questions

• RQ1 Can a single monolithic policy handle a diverse set of modular goals via a modular input encoding?
• RQ2 Does absolute learning progress-based module selection improve learning efficiency and curriculum development over random/module-specific training?
• RQ3 How does cross-module and cross-goal replay impact knowledge transfer and robustness to distractions or perturbations?
• RQ4 What self-organized developmental phases emerge when intrinsic motivation guides curriculum in modular goal RL?
• RQ5 Is the approach resilient to forgetting and sensor/body perturbations in open-ended environments?

Key findings

• m-uvfa learns multiple modular goals faster than a flat uvfa (Her) and a multi-goal module-experts baseline (mg-me).
• Intrinsic motivation based on absolute learning progress yields a self-organized succession of learning phases (developmental curriculum).
• Curious shows resilience to forgetting and sensor perturbations, recovering faster than baselines after perturbations.
• Performance degrades less with increasing distracting modules when using LP-guided replay compared to random goal selection.
• Module-based intrinsic selection helps allocate learning resources to learnable goals while disengaging from currently unsolvable or overly hard goals.