[Paper Review] Proximal Algorithms and Temporal Differences for Large Linear Systems: Extrapolation, Approximation, and Simulation

Dimitri P. Bertsekas|arXiv (Cornell University)|Oct 18, 2016

Advanced Optimization Algorithms Research54 references3 citations

TL;DR

This paper establishes a deep theoretical and algorithmic connection between proximal algorithms for solving large linear systems and temporal difference (TD) methods used in reinforcement learning and dynamic programming. By showing that TD(λ) can be interpreted as a stochastic variant of proximal iteration, the work enables direct acceleration of proximal methods via extrapolation toward TD dynamics, offering guaranteed convergence speedups without additional cost.

ABSTRACT

We consider large linear and nonlinear fixed point problems, and solution with proximal algorithms. We show that there is a close connection between two seemingly different types of methods from distinct fields: 1) Proximal iterations for linear systems of equations, which are prominent in numerical analysis and convex optimization, and 2) Temporal difference (TD) type methods, such as TD(lambda), LSTD(lambda), and LSPE(lambda), which are central in simulation-based approximate dynamic programming/reinforcement learning (DP/RL), and its recent prominent successes in large-scale game contexts, among others. One benefit of this connection is a new and simple way to accelerate the standard proximal algorithm by extrapolation towards the TD iteration, which generically has a faster convergence rate. Another benefit is the potential integration into the proximal algorithmic context of several new ideas that have emerged in the DP/RL context. We discuss some of the possibilities, and in particular, algorithms that project each proximal iterate onto the subspace spanned by a small number of basis functions, using low-dimensional calculations and simulation. A third benefit is that insights and analysis from proximal algorithms can be brought to bear on the enhancement of TD methods. The linear fixed point methodology can be extended to nonlinear fixed point problems involving a contraction, thus providing guaranteed and potentially substantial acceleration of the proximal and forward backward splitting algorithms at no extra cost. Moreover, the connection of proximal and TD methods can be extended to nonlinear (nondifferentiable) fixed point problems through new proximal-like algorithms that involve successive linearization, similar to policy iteration in DP.

Motivation & Objective

• To reveal a fundamental connection between proximal algorithms in optimization and temporal difference (TD) methods in reinforcement learning.
• To enable acceleration of proximal algorithms through extrapolation toward TD(λ) dynamics, achieving faster convergence.
• To extend the proximal framework to nonlinear fixed-point problems using successive linearization, inspired by policy iteration in dynamic programming.
• To integrate simulation-based approximation techniques from reinforcement learning into proximal algorithms for large-scale problems.
• To provide theoretical guarantees for convergence and acceleration in both exact and approximate settings, including projected and subspace-regularized variants.

Proposed method

• Formalizes the equivalence between the proximal iteration $ x_{k+1} = P^{(c)}x_k $ and the TD(λ) update via the parameter relation $ \lambda = \frac{c}{c+1} $, linking the two methods mathematically.
• Introduces a novel extrapolation scheme that accelerates the proximal algorithm by moving toward the TD(λ) iterate, ensuring guaranteed convergence speedup.
• Applies Galerkin projection to reduce dimensionality by projecting each proximal iterate onto a low-dimensional subspace spanned by basis functions, enabling scalable computation.
• Proposes proximal-like algorithms for nonlinear fixed-point problems based on successive linearization, resembling policy iteration and enabling TD-based implementations.
• Extends the framework to forward-backward splitting and proximal gradient methods, broadening applicability to composite optimization problems.
• Uses simulation-based methods to solve projected and subspace-restricted versions of the proximal update, leveraging techniques from approximate dynamic programming.

Experimental results

Research questions

• RQ1 How can proximal algorithms for large linear systems be accelerated using insights from temporal difference learning?
• RQ2 What is the precise mathematical relationship between proximal iterations and TD(λ) updates in the context of fixed-point problems?
• RQ3 Can the convergence rate of proximal methods be improved without increasing computational cost through extrapolation toward TD dynamics?
• RQ4 How can the proximal framework be extended to nonlinear fixed-point problems involving contractions, and what role does successive linearization play?
• RQ5 To what extent can simulation-based approximation techniques from reinforcement learning be integrated into proximal algorithms for large-scale problems?

Key findings

• The proximal algorithm $ x_{k+1} = P^{(c)}x_k $ is mathematically equivalent to a TD(λ) update with $ \lambda = \frac{c}{c+1} $, establishing a direct link between optimization and reinforcement learning methods.
• Extrapolation toward the TD(λ) iterate provides a guaranteed acceleration of the proximal algorithm’s convergence rate, with no additional computational cost beyond standard proximal steps.
• The projected proximal algorithm $ x_{k+1} = \Pi P^{(c)}x_k $, where $ \Pi $ projects onto a low-dimensional subspace, can be implemented via simulation-based methods, enabling scalable solutions for large systems.
• For nonlinear fixed-point problems involving contractions, the proximal framework can be extended using successive linearization, yielding new algorithms with convergence guarantees.
• The connection between proximal and TD methods generalizes to forward-backward splitting and proximal gradient methods, broadening the scope of application.
• Even when $ I - A $ is invertible but $ \sigma(A) = 1 $, the proximal equation $ x = P^{(c)}x $ remains valid and solvable via simulation, preserving the method’s robustness.