[Paper Review] Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees

Yuping Luo, Huazhe Xu+4 more|arXiv (Cornell University)|Jul 10, 2018

Reinforcement Learning in RoboticsComputer Science74 references101 citations

TL;DR

The paper proposes a meta-algorithm for model-based deep RL that guarantees monotone improvement to a local reward maximum by optimizing a data-dependent lower bound on the true value, and introduces SLBO as a practical instantiation with strong sample-efficiency.

ABSTRACT

Model-based reinforcement learning (RL) is considered to be a promising approach to reduce the sample complexity that hinders model-free RL. However, the theoretical understanding of such methods has been rather limited. This paper introduces a novel algorithmic framework for designing and analyzing model-based RL algorithms with theoretical guarantees. We design a meta-algorithm with a theoretical guarantee of monotone improvement to a local maximum of the expected reward. The meta-algorithm iteratively builds a lower bound of the expected reward based on the estimated dynamical model and sample trajectories, and then maximizes the lower bound jointly over the policy and the model. The framework extends the optimism-in-face-of-uncertainty principle to non-linear dynamical models in a way that requires extit{no explicit} uncertainty quantification. Instantiating our framework with simplification gives a variant of model-based RL algorithms Stochastic Lower Bounds Optimization (SLBO). Experiments demonstrate that SLBO achieves state-of-the-art performance when only one million or fewer samples are permitted on a range of continuous control benchmark tasks.

Motivation & Objective

• Motivate and develop a theoretical framework for model-based deep reinforcement learning with guarantees.
• Create a lower-bounding analysis that ties model error to value improvement without requiring explicit uncertainty quantification.
• Provide a practical algorithm (SLBO) that demonstrates strong sample efficiency on continuous control tasks.
• Establish discrepancy bounds that guide joint optimization of the dynamics model and policy.
• Explore Lipschitz-based and representation-invariant discrepancy bounds to improve robustness and applicability.

Proposed method

• Define a lower bound on the true value: V^{π,M*} ≥ V^{π, M̂} − D^{π_ref, δ}(M̂, π).
• Design discrepancy bounds D that satisfy: vanishing when M̂ = M*, are estimable via trajectories, and depend on a reference policy π_ref.
• Propose a meta-algorithm (Algorithm 1) that alternates optimizing (π, M̂) to maximize the lower bound while constraining policy proximity to a reference π_ref.
• Introduce and analyze telescoping lemmas to relate model prediction errors to value discrepancy (G^{π,M̂}).
• Instantiate the framework with practical bounds and a stochastic, multi-step prediction loss for model learning.
• Derive Stochastic Lower Bound Optimization (SLBO) as a simplified, implementable variant that omits certain constraints and uses a multi-step prediction loss for training the model.

Experimental results

Research questions

• RQ1 How can model-based deep RL be equipped with theoretical guarantees of improvement when optimizing over both policy and dynamics?
• RQ2 How can discrepancy bounds D^{π_ref}(M̂, π) be constructed to be estimable from trajectories and vanish when the model is accurate?
• RQ3 Does the proposed framework extend optimistic (OFU) principles to non-linear, deep dynamical models without explicit uncertainty quantification?
• RQ4 What are effective Lipschitz-based and representation-invariant discrepancy bounds, and how do they impact practical learning and performance?
• RQ5 How does the SLBO instantiation perform in sample-efficient settings on continuous control benchmarks?

Key findings

• A meta-algorithm is developed that ensures monotone improvement in V^{π,M*} under suitable discrepancy bounds and solvable optimization subproblems.
• Discrepancy bounds are derived that can be estimated from trajectories and satisfy modeling accuracy and neighborhood conditions, enabling safe joint optimization of M̂ and π.
• A telescoping lemma decomposes model-induced value discrepancy into single-step contributions, aiding theoretical guarantees.
• A representation-invariant discrepancy bound is proposed, improving robustness to state-space transformations.
• The practical SLBO variant achieves state-of-the-art performance with 1M or fewer samples on several continuous control tasks.
• The work provides theoretical foundations distinguishing model-based RL from model-free policy gradient methods in terms of error amplification and convergence behavior.