[Paper Review] A survey on intrinsic motivation in reinforcement learning

Arthur Aubret, Laëtitia Matignon+1 more|arXiv (Cornell University)|Aug 19, 2019

Reinforcement Learning in Robotics191 references92 citations

TL;DR

This survey classifies and reviews intrinsic motivation (IM) in deep RL, detailing knowledge acquisition, skill learning, and state representation, and discusses exploration, empowerment, and curriculum approaches. It compares methods that use prediction error, state novelty, information gain, and empowerment, and outlines how IM can be integrated with extrinsic rewards and goals.

ABSTRACT

The reinforcement learning (RL) research area is very active, with an important number of new contributions; especially considering the emergent field of deep RL (DRL). However a number of scientific and technical challenges still need to be addressed, amongst which we can mention the ability to abstract actions or the difficulty to explore the environment which can be addressed by intrinsic motivation (IM). In this article, we provide a survey on the role of intrinsic motivation in DRL. We categorize the different kinds of intrinsic motivations and detail for each category, its advantages and limitations with respect to the mentioned challenges. Additionnally, we conduct an in-depth investigation of substantial current research questions, that are currently under study or not addressed at all in the considered research area of DRL. We choose to survey these research works, from the perspective of learning how to achieve tasks. We suggest then, that solving current challenges could lead to a larger developmental architecture which may tackle most of the tasks. We describe this developmental architecture on the basis of several building blocks composed of a RL algorithm and an IM module compressing information.

Motivation & Objective

• Classify IM in RL and propose a two-kind orientation focused on knowledge acquisition and skill learning.
• Review how IM can be integrated with extrinsic rewards and goal spaces to guide learning.
• Analyze the role of IM in improving state representation, exploration, and empowerment.
• Summarize alternative curricula and goal-generation approaches within IM-driven RL.

Proposed method

• Present a classification of IM in DRL and map it to Section 6 Knowledge acquisition and Section 7 Skill learning.
• Survey core IM techniques: prediction error, state novelty, information gain, and empowerment.
• Discuss integration approaches where r = α r_int + β r_ext or V(s) = α V_int(s) + β V_ext(s).
• Describe exploration mechanisms that address white-noise, including ICM, EMI, pseudo-count/density-based methods, and RND.
• Explain empowerment as a mutual-information objective and how it can be computed or bounded in practice.
• Outline how IM informs state representation learning and the construction of goal spaces for skills.

Experimental results

Research questions

• RQ1 How can intrinsic motivation be systematically classified in deep RL?
• RQ2 What are the main IM mechanisms (prediction error, state novelty, information gain, empowerment) and how do they compare in exploration tasks?
• RQ3 How can IM be used to learn and transfer skills (options) and build goal spaces without supervision?
• RQ4 When and how should intrinsic rewards be combined with extrinsic rewards or used to shape representations and curricula?
• RQ5 What are the practical challenges (computation, scalability, long-term control) in applying IM, and how can they be mitigated?

Key findings

• Prediction-error based, state-novelty based, and information-gain based IM approaches each address exploration but with different computational costs and suitability to environments.
• Intrinsic curiosity modules (ICM) and related methods help avoid white-noise problems by focusing on controllable aspects of the environment.
• Random network distillation (RND) and density/pseudo-count methods improve exploration in sparse-reward tasks like Montezuma’s revenge, but scaling and generalization remain concerns.
• Empowerment-based IM encourages learning controllable behaviors and can stabilize multi-agent training and lifelong skill diversification, though it is computationally intensive.
• State representation learning benefits from IM by shaping the distribution of interactions, enabling disentangled representations and more effective generalization.
• Skill learning with intrinsic rewards enables the discovery and reuse of task-agnostic skills (options), and goal-space construction from state representations can facilitate hierarchical RL and curriculum learning.