[Paper Review] Incremental Truncated LSTD

Clement Gehring, Yangchen Pan+1 more|arXiv (Cornell University)|Nov 26, 2015

Reinforcement Learning in Robotics31 references3 citations

TL;DR

This paper proposes t-LSTD(λ), an incremental low-rank approximation of Least-Squares Temporal Difference learning that reduces computational and storage complexity from O(d²) to O(dr) by maintaining a truncated singular value decomposition (SVD) of the LSTD matrix. The method achieves near-LSTD sample efficiency with TD-like computational efficiency, outperforming both TD and iLSTD in benchmark and high-dimensional domains while eliminating the need for step-size tuning.

ABSTRACT

Balancing between computational efficiency and sample efficiency is an important goal in reinforcement learning. Temporal difference (TD) learning algorithms stochastically update the value function, with a linear time complexity in the number of features, whereas least-squares temporal difference (LSTD) algorithms are sample efficient but can be quadratic in the number of features. In this work, we develop an efficient incremental low-rank LSTD(λ) algorithm that progresses towards the goal of better balancing computation and sample efficiency. The algorithm reduces the computation and storage complexity to the number of features times the chosen rank parameter while summarizing past samples efficiently to nearly obtain the sample complexity of LSTD. We derive a simulation bound on the solution given by truncated low-rank approximation, illustrating a bias- variance trade-off dependent on the choice of rank. We demonstrate that the algorithm effectively balances computational complexity and sample efficiency for policy evaluation in a benchmark task and a high-dimensional energy allocation domain.

Motivation & Objective

• To address the trade-off between computational efficiency and sample efficiency in value function approximation for reinforcement learning.
• To reduce the O(d²) storage and computation of standard LSTD to O(dr) using low-rank approximation via truncated SVD.
• To enable efficient, incremental updates with regularization through rank truncation, improving numerical stability and adaptability.
• To empirically validate the effectiveness of low-rank approximation in high-dimensional feature spaces, such as tile coding and RBFs.
• To demonstrate that t-LSTD(λ) outperforms both TD and iLSTD in sample efficiency and computational efficiency without requiring step-size tuning.

Proposed method

• The algorithm maintains an incremental truncated SVD of the LSTD matrix A, updating the low-rank approximation of the feature covariance matrix in real time.
• It uses a rank parameter r to control the trade-off between approximation accuracy and computational cost, truncating singular values below the r-th largest.
• The solution vector w is computed via a closed-form update using the truncated SVD, enabling O(dr + r³) computation per step.
• The method supports efficient down-weighting of past samples via a decay factor βt ∈ (0,1), enabling adaptation in non-stationary environments.
• It incorporates λ-return for eligibility traces, generalizing to λ > 0, and uses a fixed heuristic for thresholding in the initial implementation.
• The algorithm is derived to maintain the same solution structure as LSTD but with reduced dimensionality through SVD truncation.

Experimental results

Research questions

• RQ1 Can low-rank approximation of the LSTD matrix significantly reduce computational and storage costs while preserving sample efficiency?
• RQ2 How does the choice of rank r affect the bias-variance trade-off in the value function approximation?
• RQ3 Does t-LSTD(λ) outperform TD and iLSTD in both sample efficiency and computational efficiency in high-dimensional feature spaces?
• RQ4 Can t-LSTD(λ) be used effectively for online, incremental learning with minimal hyperparameter tuning?
• RQ5 What is the impact of truncating singular values on the convergence and stability of the value function estimate?

Key findings

• t-LSTD with r = 40 outperformed both TD and iLSTD in the Mountain Car domain despite d = 40,000 features, achieving faster and more stable learning.
• The algorithm achieved near-LSTD sample efficiency with O(dr) complexity, significantly reducing storage and computation compared to full LSTD.
• t-LSTD eliminated the need for step-size tuning, unlike iLSTD, which was highly sensitive to step-size selection and prone to divergence.
• The method demonstrated strong performance in a high-dimensional energy allocation domain, showing scalability and robustness.
• Empirical results confirmed that low-rank structure exists in common feature representations like tile coding and RBFs, validating the use of SVD truncation.
• The simulation error bound derived in the paper confirms a clear bias-variance trade-off dependent on the rank r, with lower r increasing bias but improving stability and speed.