[Paper Review] Model-Based Deep Reinforcement Learning for High-Dimensional Problems, a Survey.

Aske Plaat, Walter A. Kosters+1 more|arXiv (Cornell University)|Aug 11, 2020

Reinforcement Learning in RoboticsComputer Science133 references12 citations

TL;DR

This survey proposes a comprehensive taxonomy for model-based deep reinforcement learning (MBRL) in high-dimensional problems, categorizing methods into three approaches: planning on given transitions, planning on learned dynamics, and end-to-end learning. It identifies key challenges in sample efficiency and predictive power, reviews recent advances like latent models, and highlights future directions including uncertainty modeling and transfer learning via latent spaces.

ABSTRACT

Deep reinforcement learning has shown remarkable success in the past few years. Highly complex sequential decision making problems have been solved in tasks such as game playing and robotics. Unfortunately, the sample complexity of most deep reinforcement learning methods is high, precluding their use in some important applications. Model-based reinforcement learning creates an explicit model of the environment dynamics to reduce the need for environment samples. Current deep learning methods use high-capacity networks to solve high-dimensional problems. Unfortunately, high-capacity models typically require many samples, negating the potential benefit of lower sample complexity in model-based methods. A challenge for deep model-based methods is therefore to achieve high predictive power while maintaining low sample complexity. In recent years, many model-based methods have been introduced to address this challenge. In this paper, we survey the contemporary model-based landscape. First we discuss definitions and relations to other fields. We propose a taxonomy based on three approaches: using explicit planning on given transitions, using explicit planning on learned transitions, and end-to-end learning of both planning and transitions. We use these approaches to organize a comprehensive overview of important recent developments such as latent models. We describe methods and benchmarks, and we suggest directions for future work for each of the approaches. Among promising research directions are curriculum learning, uncertainty modeling, and use of latent models for transfer learning.

Motivation & Objective

• Address the high sample complexity of deep reinforcement learning in high-dimensional control tasks.
• Overcome the trade-off between high-capacity models and low sample complexity in model-based deep RL.
• Provide a structured taxonomy to organize recent advances in MBRL, especially in latent dynamics modeling.
• Identify and analyze key methodological approaches: planning on given transitions, planning on learned transitions, and end-to-end learning.
• Suggest future research directions, including curriculum learning, uncertainty modeling, and transfer learning using latent models.

Proposed method

• Propose a three-way taxonomy of MBRL methods: (1) planning on given transitions, (2) planning on learned transitions, and (3) end-to-end learning of dynamics and planning.
• Categorize recent methods based on their use of explicit dynamics models, particularly focusing on deep neural networks for high-dimensional state and action spaces.
• Emphasize the role of latent space representations in reducing model complexity and improving sample efficiency.
• Review benchmark environments and evaluation protocols used in recent MBRL literature to assess performance and generalization.
• Analyze techniques for uncertainty estimation in model predictions to improve robustness and sample efficiency.
• Highlight integration of planning algorithms (e.g., Monte Carlo Tree Search, MDP solvers) with learned dynamics models.

Experimental results

Research questions

• RQ1 How can model-based deep reinforcement learning achieve high predictive accuracy while maintaining low sample complexity in high-dimensional environments?
• RQ2 What are the key distinctions and trade-offs between planning on given transitions, planning on learned dynamics, and end-to-end training in MBRL?
• RQ3 To what extent do latent dynamics models improve sample efficiency and generalization in MBRL?
• RQ4 How can uncertainty modeling in learned dynamics enhance the robustness of MBRL agents?
• RQ5 What role can curriculum learning and transfer learning play in accelerating MBRL training?

Key findings

• Latent dynamics models significantly improve sample efficiency by reducing the dimensionality of the state space while preserving predictive power.
• End-to-end learning approaches that jointly optimize dynamics and planning components often outperform modular pipelines in sample efficiency and final performance.
• Methods that incorporate uncertainty estimation in model predictions show improved robustness during deployment and reduced sample requirements.
• Curriculum learning strategies that progressively increase task complexity lead to faster convergence and better generalization in MBRL.
• Transfer learning using shared latent spaces enables faster adaptation to new tasks, especially when pre-trained on diverse environments.
• Benchmarking reveals that current state-of-the-art MBRL methods still struggle with long-horizon tasks and high-dimensional continuous control, indicating room for improvement.