[Paper Review] Model-Based Reinforcement Learning with Value-Targeted Regression

Alex Ayoub, Zeyu Jia+3 more|arXiv (Cornell University)|Jun 1, 2020

Advanced Bandit Algorithms Research44 references71 citations

TL;DR

The paper introduces UCRL-VTR, a model-based RL algorithm that uses value-targeted regression to construct confidence sets and perform optimistic planning, achieving regret bounds that scale with model complexity rather than state/action space size, including a bound for linear mixtures.

ABSTRACT

This paper studies model-based reinforcement learning (RL) for regret minimization. We focus on finite-horizon episodic RL where the transition model $P$ belongs to a known family of models $\mathcal{P}$, a special case of which is when models in $\mathcal{P}$ take the form of linear mixtures: $P_θ = \sum_{i=1}^{d} θ_{i}P_{i}$. We propose a model based RL algorithm that is based on optimism principle: In each episode, the set of models that are `consistent' with the data collected is constructed. The criterion of consistency is based on the total squared error of that the model incurs on the task of predicting \emph{values} as determined by the last value estimate along the transitions. The next value function is then chosen by solving the optimistic planning problem with the constructed set of models. We derive a bound on the regret, which, in the special case of linear mixtures, the regret bound takes the form $ ilde{\mathcal{O}}(d\sqrt{H^{3}T})$, where $H$, $T$ and $d$ are the horizon, total number of steps and dimension of $θ$, respectively. In particular, this regret bound is independent of the total number of states or actions, and is close to a lower bound $Ω(\sqrt{HdT})$. For a general model family $\mathcal{P}$, the regret bound is derived using the notion of the so-called Eluder dimension proposed by Russo & Van Roy (2014).

Motivation & Objective

• Motivate regret minimization in online model-based RL under a known family of transition models P.
• Propose value-targeted regression to build data-consistent confidence sets for P.
• Develop an optimistic planning-based algorithm (UCRL-VTR) leveraging these sets.
• Provide theoretical regret bounds and evaluate the approach empirically.

Proposed method

• Define episodic MDPs with a known model family P and consider linear mixture models P = sum_j theta_j P_j.
• Introduce value-targeted regression to form a regression loss L_{k+1}(P, P̂_{k+1}) based on predicted values V_{h+1,k} and observed targets y_{h,k}.
• Construct confidence sets B_k from the regression loss, i.e., B_{k+1} = {P' ∈ P : L_{k+1}(P', P̂_{k+1}) ≤ β_{k+1}}.
• In each episode, perform optimistic planning over B_k to choose P_k that maximizes V^{*}_{P',1}(s_1^k), then execute the induced policy and update the value targets.
• Provide regret bounds in terms of Eluder dimension and covering numbers; specialize to linear mixtures to obtain R_K = Ō(d √(H^3 K)) and a lower bound Ω(√(HdK)).
• Discuss implementation considerations and connections to MuZero.

Experimental results

Research questions

• RQ1 Can value-targeted regression yield sublinear regret for model-based RL across general model classes P?
• RQ2 How do regret bounds depend on the complexity of P, e.g., Eluder dimension, and on noise/nonstationarity in value targets?
• RQ3 What are the benefits and limitations of optimistic planning with a value-targeted confidence set compared to traditional model-based approaches?
• RQ4 How does the approach specialize to linear mixture models in terms of regret scaling?
• RQ5 How does the method compare empirically to other model-based RL methods and to value-targeted regression variants?

Key findings

• For linear mixture models, the algorithm achieves a regret bound of Ō(d √(H^3 T)).
• In the general model class setting, the regret is bounded via the Eluder dimension of the function class defined by value targets.
• The upper bound is independent of the size of the state or action spaces, and near a lower bound of Ω(√(HdT)) in the linear case.
• Value-targeted regression focuses model learning on task-relevant dynamics, potentially improving efficiency over likelihood-based regression.
• Experiments show that value-targeted regression with optimistic planning is effective, while removing optimism or value-targeted regression degrades performance.
• The work connects to MuZero, which independently uses value-targeted regression for model construction.