[Paper Review] Rate of Convergence and Error Bounds for LSTD($λ$)

Manel Tagorti, Bruno Scherrer|arXiv (Cornell University)|May 13, 2014

Markov Chains and Monte Carlo Methods11 references3 citations

TL;DR

This paper establishes high-probability error bounds and a rate of convergence for LSTD(λ), a least-squares temporal-difference algorithm with eligibility traces, under β-mixing assumptions. It proves a convergence rate of Õ(1/√n) for any λ ∈ (0,1), extending and improving prior bounds for λ = 0 and providing insight into the optimal choice of λ based on the quality of the linear function approximation and sample size.

ABSTRACT

We consider LSTD($λ$), the least-squares temporal-difference algorithm with eligibility traces algorithm proposed by Boyan (2002). It computes a linear approximation of the value function of a fixed policy in a large Markov Decision Process. Under a $β$-mixing assumption, we derive, for any value of $λ\in (0,1)$, a high-probability estimate of the rate of convergence of this algorithm to its limit. We deduce a high-probability bound on the error of this algorithm, that extends (and slightly improves) that derived by Lazaric et al. (2012) in the specific case where $λ=0$. In particular, our analysis sheds some light on the choice of $λ$ with respect to the quality of the chosen linear space and the number of samples, that complies with simulations.

Motivation & Objective

• To close the gap in finite-sample error bounds for LSTD(λ) when λ > 0, despite known asymptotic convergence.
• To extend the high-probability error bound previously derived by Lazaric et al. (2012) for λ = 0 to general λ ∈ (0,1).
• To provide theoretical justification for the choice of λ in terms of approximation quality and sample efficiency.
• To analyze the rate of convergence of LSTD(λ) to its limit under β-mixing conditions.
• To offer practical guidance on balancing approximation error and sample complexity via λ selection.

Proposed method

• Derives a high-probability bound on the estimation error of LSTD(λ) using concentration inequalities under β-mixing assumptions.
• Applies a martingale-type argument and Bernstein-type inequalities to control the deviation of empirical averages from their expectations.
• Uses the structure of eligibility traces to express the LSTD(λ) update as a weighted sum of Bellman operator applications.
• Establishes a rate of convergence of Õ(1/√n) by bounding the deviation of the empirical estimate from the true solution.
• Introduces a coupling argument involving a mixing time m = ⌈log(n−1)/log(1/(λγ))⌉ to decouple dependence in the trajectory.
• Employs a chaining argument and exponential moment bounds to control the norm of the empirical error vector.

Experimental results

Research questions

• RQ1 What is the finite-sample convergence rate of LSTD(λ) for λ ∈ (0,1) under β-mixing?
• RQ2 How does the error bound of LSTD(λ) depend on the choice of λ, the quality of the linear function approximation, and the number of samples?
• RQ3 Can the high-probability error bound for λ = 0 be extended to λ > 0, and if so, how does it compare?
• RQ4 What is the optimal trade-off between approximation error and estimation error controlled by λ?
• RQ5 How does the mixing time of the Markov chain affect the convergence rate of LSTD(λ)?

Key findings

• LSTD(λ) converges to its limit at a rate of Õ(1/√n) for any λ ∈ (0,1), under β-mixing assumptions.
• The high-probability error bound for LSTD(λ) is of the form Õ(1/√n), extending and slightly improving upon the bound for λ = 0.
• The choice of λ influences the trade-off between approximation error (via the projection type) and estimation error (via variance control).
• The analysis shows that increasing λ improves the approximation quality by moving from oblique to orthogonal projection, which reduces the approximation error.
• The bound depends on the mixing coefficient β and the feature space dimension d, with explicit dependence on log(1/δ) and the mixing time.
• The optimal λ balances the approximation error (decreasing with λ) and the estimation error (increasing with λ), and the bound quantifies this trade-off.