[Paper Review] Variance-reduced $Q$-learning is minimax optimal

Martin J. Wainwright|arXiv (Cornell University)|Jun 11, 2019

Machine Learning and Algorithms40 references44 citations

TL;DR

Introduces a variance-reduced variant of Q-learning for finite MDPs with γ-discounting and proves it achieves minimax-optimal sample complexity up to log factors in the discount complexity. It shows substantial improvement over ordinary Q-learning in dependence on 1/(1−γ).

ABSTRACT

We introduce and analyze a form of variance-reduced $Q$-learning. For $γ$-discounted MDPs with finite state space $\mathcal{X}$ and action space $\mathcal{U}$, we prove that it yields an $ε$-accurate estimate of the optimal $Q$-function in the $\ell_\infty$-norm using $\mathcal{O} \left(\left(\frac{D}{ ε^2 (1-γ)^3} ight) \; \log \left( \frac{D}{(1-γ)} ight) ight)$ samples, where $D = |\mathcal{X}| imes |\mathcal{U}|$. This guarantee matches known minimax lower bounds up to a logarithmic factor in the discount complexity. In contrast, our past work shows that ordinary $Q$-learning has worst-case quartic scaling in the discount complexity.

Motivation & Objective

• Motivate the study of variance-reduced methods in Q-learning for γ-discounted finite MDPs.
• Propose a practical variance-reduced Q-learning algorithm inspired by SVRG.
• Establish non-asymptotic, high-probability convergence guarantees.
• Show minimax-optimal sample complexity up to logarithmic factors in the discount complexity 1/(1−γ).
• Compare with prior Q-learning results and identify improvements in dependence on (1−γ).

Proposed method

• Define a variance-reduced Q-learning operator that uses a Monte Carlo approximation of the Bellman update with unbiased recentering.
• Structure the algorithm into epochs with a variance-reduced update θk+1 = (1−λk)θk + λk(bTk(θk) − bTk(θ) + eTN(θ)) where eTN(θ) is an unbiased estimate of T(θ).
• Use epoch-length K and recentering sample sizes Nm to control bias and variance, with step sizes λk = 1/(1+(1−γ)k).
• Provide RunEpoch and overall Algorithm Variance-reduced Q-learning with M epochs, each of length K and recentering samples Nm.
• Derive parameter choices: K = c1 log(8MD(1−γ)−δ)/( (1−γ)3 ), Nm = c2 4m log(8MD/δ)/( (1−γ)2 ).
• Prove geometric convergence over epochs and give explicit total-sample bounds culminating in minimax-optimal results up to logarithmic factors.

Experimental results

Research questions

• RQ1 Can a simple variance-reduction extension of Q-learning achieve minimax-optimal sample complexity for estimating the optimal Q-function in ℓ∞-norm?
• RQ2 How should epoch structure, recentering, and step sizes be designed to balance bias and variance in variance-reduced Q-learning?
• RQ3 What are the precise non-asymptotic, high-probability guarantees (convergence rate and sample complexity) for the proposed method?
• RQ4 How does the proposed method compare to existing Q-learning and Q-value iteration approaches in terms of dependence on (1−γ)?

Key findings

• The variance-reduced Q-learning algorithm achieves geometric convergence over epochs with high probability.
• The final error after M epochs satisfies ∥θM − θ∗∥∞ ≤ ∥σ(θ∗)∥∞ + ∥θ∗∥∞(1−γ)2M with probability at least 1−δ.
• The total sample complexity to achieve ϵ-accuracy is bounded by a log-factor dependent expression, improving over ordinary Q-learning and matching minimax lower bounds up to log factors.
• In the worst-case over γ-discounted MDPs with rmax-bounded rewards, the method attains the cubic 1/(1−γ)3 scaling, matching known minimax lower bounds up to logarithmic factors (Proposition 1).
• A refined analysis shows that starting from an initialization within rmax√(1−γ) of θ∗ yields minimax-optimal sample complexity (Proposition 1).