[Paper Review] Model-based Reinforcement Learning: A Survey

Thomas M. Moerland, Joost Broekens+2 more|arXiv (Cornell University)|Jun 30, 2020

Complex Systems and Decision MakingDecision Sciences82 citations

TL;DR

This paper surveys model-based reinforcement learning, detailing dynamics model learning and planning-learning integration, including implicit model-based RL and potential benefits.

ABSTRACT

Sequential decision making, commonly formalized as Markov Decision Process (MDP) optimization, is a important challenge in artificial intelligence. Two key approaches to this problem are reinforcement learning (RL) and planning. This paper presents a survey of the integration of both fields, better known as model-based reinforcement learning. Model-based RL has two main steps. First, we systematically cover approaches to dynamics model learning, including challenges like dealing with stochasticity, uncertainty, partial observability, and temporal abstraction. Second, we present a systematic categorization of planning-learning integration, including aspects like: where to start planning, what budgets to allocate to planning and real data collection, how to plan, and how to integrate planning in the learning and acting loop. After these two sections, we also discuss implicit model-based RL as an end-to-end alternative for model learning and planning, and we cover the potential benefits of model-based RL. Along the way, the survey also draws connections to several related RL fields, like hierarchical RL and transfer learning. Altogether, the survey presents a broad conceptual overview of the combination of planning and learning for MDP optimization.

Motivation & Objective

• Explain the integration of planning and learning in the model-based RL paradigm.
• Categorize how dynamics models are learned and the challenges involved (stochasticity, uncertainty, partial observability, non-stationarity).
• Systematically discuss how planning is integrated with learning and when to start planning within the learning loop.
• Introduce implicit model-based RL as an end-to-end alternative and discuss potential benefits.

Proposed method

• Define model-based RL and distinguish it from planning and model-free RL based on access to dynamics and global solutions.
• Provide a three-way categorization of planning-learning integration: model-based RL with a learned model, model-based RL with a known model, and planning over a learned model.
• Review dynamics model learning, including forward, backward, and inverse models, and discuss estimation methods (parametric vs non-parametric, exact vs approximate).
• Discuss region of model validity (global vs local) and how this affects planning and learning.
• Address challenges in modeling: stochasticity, uncertainty, partial observability, non-stationarity, multi-step prediction, state and temporal abstraction.
• Outline approaches to integration of planning and learning, including planning start state, planning budget, planning methods, and end-to-end learning variants.

Experimental results

Research questions

• RQ1 How can planning and learning be integrated in MDP optimization to form effective model-based RL algorithms?
• RQ2 What are the key challenges in dynamics model learning and how can they be addressed (stochasticity, uncertainty, partial observability, non-stationarity)?
• RQ3 What are the trade-offs between global vs local model validity and their impact on planning and data efficiency?
• RQ4 How should planning be organized within the learning loop (when to plan, how much data to collect, how to plan)?

Key findings

• Model-based RL combines a dynamics model with a global solution to improve sample efficiency.
• Forward models with parametric approximation and global coverage are common in practice.
• Partial observability and non-stationarity require specialized techniques like belief states, recurrence, and partial models.
• Multi-step prediction challenges motivate using multi-step losses or dedicated n-step models to improve long-horizon accuracy.
• Planning can occur over learned models, with learning of the policy/value function, or both; planning over a learned model is not considered model-based RL if no global solution is learned.
• Implicit model-based RL offers end-to-end alternatives that still leverage planning and modeling benefits.