[Paper Review] Reinforcement Learning with General Value Function Approximation: Provably Efficient Approach via Bounded Eluder Dimension

Ruosong Wang, Ruslan Salakhutdinov+1 more|arXiv (Cornell University)|May 21, 2020

Reinforcement Learning in Robotics67 references30 citations

TL;DR

Proposes a provably efficient Q-learning algorithm for RL with general value function approximation, with regret depending on the eluder dimension and log-covering numbers of the function class, without assuming a specific model. It uses a stable UCB-like bonus and a data-subsampling scheme for efficiency.

ABSTRACT

Value function approximation has demonstrated phenomenal empirical success in reinforcement learning (RL). Nevertheless, despite a handful of recent progress on developing theory for RL with linear function approximation, the understanding of general function approximation schemes largely remains missing. In this paper, we establish a provably efficient RL algorithm with general value function approximation. We show that if the value functions admit an approximation with a function class $\mathcal{F}$, our algorithm achieves a regret bound of $\widetilde{O}(\mathrm{poly}(dH)\sqrt{T})$ where $d$ is a complexity measure of $\mathcal{F}$ that depends on the eluder dimension [Russo and Van Roy, 2013] and log-covering numbers, $H$ is the planning horizon, and $T$ is the number interactions with the environment. Our theory generalizes recent progress on RL with linear value function approximation and does not make explicit assumptions on the model of the environment. Moreover, our algorithm is model-free and provides a framework to justify the effectiveness of algorithms used in practice.

Motivation & Objective

• Motivate and address RL with general value function approximation beyond linear settings.
• Develop a provably efficient, model-free Q-learning algorithm that works with a general function class F.
• Characterize the algorithm's regret in terms of eluder dimension and covering numbers of F and the state-action space.

Proposed method

• Define a general function class F for Q-function approximation and assume Bellman-backup compatibility: for any V, there exists f_V in F with f_V(s,a) = r(s,a) + sum_{s'} P(s'|s,a) V(s').
• Iteratively compute Q^k_h by least-squares fits over a replay buffer and add a Stable UCB bonus b^k_h to encourage exploration.
• Use a data-driven confidence region F^k_h and its width w(F^k_h, s,a) as the bonus, ensuring Q^k_h is an overestimate with high probability.
• Introduce stability via importance-sampling based sensitivity sampling to subsample the dataset and control the complexity of the bonus.
• Provide Algorithm 1 (F-LSVI) with a construction of Q-values and greedy policies, and Algorithm 3 (Bonus) to generate the stable bonus.
• Quantify the regret as a function of the eluder dimension dim_E(F, δ/T^3) and covering numbers N(F, δ/T^2) and N(S×A, δ/T) under Assumption 1.

Experimental results

Research questions

• RQ1 Can RL with general function approximation achieve provable efficiency without model-based assumptions?
• RQ2 How can eluder dimension and covering numbers of the value function class govern the regret of a model-free, function-approximation RL algorithm?
• RQ3 What are practical mechanisms (like stable bonuses and data subsampling) to ensure exploration and computational efficiency?
• RQ4 How does the proposed approach relate to and generalize existing results for linear and generalized linear function approximation?

Key findings

• The algorithm achieves a regret bound with dependence on H^2 T and a complexity term involving dim_E(F, δ/T^3) and logarithmic factors of covering numbers.
• For tabular RL, the bound reduces to a form comparable with existing tabular results, while acknowledging a worse naive bound due to generality.
• When F is d-dimensional linear or generalized linear, dim_E(F, ε) = O(d log(1/ε)) or analogous bounds, leading to a regret that scales with the corresponding complexity terms and logs.
• The approach generalizes RL with linear function approximation and provides a model-free framework that does not assume explicit environment dynamics.
• The Stable Bonus Function and Sensitivity-Sampling enable computationally feasible implementations by controlling dataset size and maintaining confident over-estimations of Q.