[Paper Review] Nearly Minimax Optimal Reinforcement Learning for Linear Markov Decision Processes

Jiafan He, Heyang Zhao+2 more|arXiv (Cornell University)|Dec 12, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes LSVI-UCB++, the first computationally efficient reinforcement learning algorithm for linear Markov decision processes that achieves nearly minimax optimal regret of $\widetilde{O}(d\sqrt{H^3K})$. It combines a variance-aware weighted ridge regression with a rare-switching policy to improve estimation accuracy and reduce regret, matching the theoretical lower bound up to logarithmic factors.

ABSTRACT

We study reinforcement learning (RL) with linear function approximation. For episodic time-inhomogeneous linear Markov decision processes (linear MDPs) whose transition probability can be parameterized as a linear function of a given feature mapping, we propose the first computationally efficient algorithm that achieves the nearly minimax optimal regret $ ilde O(d\sqrt{H^3K})$, where $d$ is the dimension of the feature mapping, $H$ is the planning horizon, and $K$ is the number of episodes. Our algorithm is based on a weighted linear regression scheme with a carefully designed weight, which depends on a new variance estimator that (1) directly estimates the variance of the optimal value function, (2) monotonically decreases with respect to the number of episodes to ensure a better estimation accuracy, and (3) uses a rare-switching policy to update the value function estimator to control the complexity of the estimated value function class. Our work provides a complete answer to optimal RL with linear MDPs, and the developed algorithm and theoretical tools may be of independent interest.

Motivation & Objective

• To close the gap between existing computationally efficient RL algorithms and the minimax optimal regret bound for linear MDPs.
• To develop a method that achieves nearly optimal regret without relying on expensive integration or sampling oracles required by prior linear mixture MDP approaches.
• To design a variance estimator that directly targets the true value function's variance, enabling tighter confidence sets.
• To ensure computational efficiency while maintaining uniform control over the value function estimator's complexity via a rare-switching update strategy.

Proposed method

• Proposes a weighted ridge regression scheme with a novel variance-aware weight function that improves estimation accuracy.
• Introduces a new variance estimator that directly estimates the variance of the true optimal value function, reducing overestimation risk.
• Employs a monotonically decreasing value function estimator that ensures improved estimation accuracy over time.
• Uses a rare-switching policy to update the value function estimator, controlling the complexity of the estimated function class.
• Applies a confidence ellipsoid framework based on high-probability bounds for linear regression, leveraging matrix concentration inequalities.
• Integrates a novel regularized regression procedure that ensures uniform control over estimation error across episodes.

Experimental results

Research questions

• RQ1 Can a computationally efficient RL algorithm achieve nearly minimax optimal regret in linear MDPs?
• RQ2 How can variance estimation be improved to reduce regret beyond existing LSVI-UCB-style algorithms?
• RQ3 Can the regret bound be tightened to match the known lower bound of $O(d\sqrt{H^3K})$ without relying on generative models or expensive oracles?
• RQ4 What role does a monotonically decreasing value function estimator play in improving estimation accuracy and regret bounds?
• RQ5 How can the complexity of the estimated value function class be controlled to ensure uniform convergence?

Key findings

• The proposed LSVI-UCB++ algorithm achieves a regret bound of $\widetilde{O}(d\sqrt{H^3K})$, matching the known lower bound up to logarithmic factors.
• The algorithm improves upon the $\widetilde{O}(\sqrt{d^3H^4K})$ regret of Jin et al. (2020) by reducing the dependence on $H$ via a variance-aware regression scheme.
• The new variance estimator directly targets the true value function’s variance, leading to a tighter confidence set than previous methods.
• The rare-switching policy ensures that the value function estimator remains computationally efficient while maintaining uniform estimation accuracy.
• The method avoids reliance on integration or sampling oracles, making it applicable in general linear MDPs where such oracles are intractable.
• The theoretical framework resolves a technical flaw in a concurrent work (Hu et al., 2022), validating the correctness of the approach through rigorous analysis.