[Paper Review] Finite-Sample Analysis of Off-Policy TD-Learning via Generalized Bellman Operators

Zaiwei Chen, Siva Theja Maguluri+2 more|arXiv (Cornell University)|Jun 24, 2021

Reinforcement Learning in Robotics37 references4 citations

TL;DR

This paper provides the first finite-sample analysis of off-policy TD-learning using generalized Bellman operators, proving that the operator is a contraction under weighted ℓp-norms for all p ∈ [1, ∞) with a uniform contraction factor. The framework yields the first-known finite-sample bounds for Qπ(λ), Tree-Backup(λ), and Retrace(λ), and improves existing bounds for Q-trace, revealing explicit bias-variance trade-offs across algorithms.

ABSTRACT

In temporal difference (TD) learning, off-policy sampling is known to be more practical than on-policy sampling, and by decoupling learning from data collection, it enables data reuse. It is known that policy evaluation (including multi-step off-policy importance sampling) has the interpretation of solving a generalized Bellman equation. In this paper, we derive finite-sample bounds for any general off-policy TD-like stochastic approximation algorithm that solves for the fixed-point of this generalized Bellman operator. Our key step is to show that the generalized Bellman operator is simultaneously a contraction mapping with respect to a weighted $\ell_p$-norm for each $p$ in $[1,\infty)$, with a common contraction factor. Off-policy TD-learning is known to suffer from high variance due to the product of importance sampling ratios. A number of algorithms (e.g. $Q^π(λ)$, Tree-Backup$(λ)$, Retrace$(λ)$, and $Q$-trace) have been proposed in the literature to address this issue. Our results immediately imply finite-sample bounds of these algorithms. In particular, we provide first-known finite-sample guarantees for $Q^π(λ)$, Tree-Backup$(λ)$, and Retrace$(λ)$, and improve the best known bounds of $Q$-trace in [19]. Moreover, we show the bias-variance trade-offs in each of these algorithms.

Motivation & Objective

• To establish finite-sample convergence guarantees for general off-policy TD-learning algorithms that solve generalized Bellman equations.
• To address the high variance issue in off-policy TD-learning caused by importance sampling ratios.
• To unify and analyze multiple existing algorithms—Qπ(λ), Tree-Backup(λ), Retrace(λ), and Q-trace—under a single theoretical framework.
• To reveal the bias-variance trade-offs inherent in these algorithms through explicit sample complexity bounds.

Proposed method

• Model off-policy TD-learning as a Markovian stochastic approximation algorithm targeting the fixed point of a generalized Bellman operator.
• Define the generalized Bellman operator B(Q) = T(H(Q) - Q) + Q, where H is the target operator and T controls contraction properties.
• Prove that the generalized Bellman operator is a contraction mapping under weighted ℓp-norms for all p ∈ [1, ∞), with a common contraction factor independent of p.
• Use the uniform contraction property across ℓp-norms to derive tight finite-sample bounds scaling as Õ(ε⁻²) for accuracy ε.
• Construct a dominating stochastic matrix M'' with strictly positive entries to ensure existence of a positive stationary distribution, enabling norm contraction analysis.
• Apply the results to derive sample complexity bounds for Qπ(λ), Tree-Backup(λ), Retrace(λ), and Q-trace, improving upon prior work.

Experimental results

Research questions

• RQ1 Can a unified theoretical framework be developed to analyze finite-sample convergence of off-policy TD-learning algorithms?
• RQ2 What conditions ensure that the generalized Bellman operator is a contraction under weighted ℓp-norms for all p ∈ [1, ∞)?
• RQ3 How do the bias and variance of off-policy algorithms like Qπ(λ), Tree-Backup(λ), and Retrace(λ) trade off in terms of sample complexity?
• RQ4 Can tighter finite-sample bounds be established for Q-trace compared to prior work, especially in terms of state-action space dependence?
• RQ5 What is the role of the generalized importance sampling ratios in determining the convergence rate of off-policy TD-learning?

Key findings

• The generalized Bellman operator is a contraction under weighted ℓp-norms for all p ∈ [1, ∞), with a uniform contraction factor, enabling tight finite-sample analysis.
• The proposed framework yields the first-known finite-sample bounds for Qπ(λ), Tree-Backup(λ), and Retrace(λ), resolving a gap in the literature.
• For Q-trace, the sample complexity is improved by a factor of at least |S||A| compared to the prior best result in [19], reducing dependency on state-action space size.
• The finite-sample bounds explicitly reveal the bias-variance trade-offs inherent in each algorithm, with variance reduction achieved at the cost of increased bias.
• The contraction property is established via a construction of a strictly positive stochastic matrix M'' dominating the transition matrix, ensuring a unique positive stationary distribution.
• The sample complexity scales as Õ(ε⁻²), where ε is the target accuracy, with problem-dependent factors involving generalized importance sampling ratios.