[Paper Review] Predictive State Temporal Difference Learning

Byron Boots, Geoffrey J. Gordon|arXiv (Cornell University)|Oct 30, 2010

Reinforcement Learning in Robotics34 references19 citations

TL;DR

This paper introduces Predictive State Temporal Difference (PSTD) learning, a novel algorithm that combines linear temporal difference learning with subspace identification to automatically compress a large set of features into a minimal, predictive subspace. PSTD achieves statistically consistent value function estimation by preserving only predictive information, outperforming LSTD and LARS-TD on a high-dimensional optimal stopping problem for financial derivatives, reducing error by 1.24 percentage points over prior best results.

ABSTRACT

We propose a new approach to value function approximation which combines linear temporal difference reinforcement learning with subspace identification. In practical applications, reinforcement learning (RL) is complicated by the fact that state is either high-dimensional or partially observable. Therefore, RL methods are designed to work with features of state rather than state itself, and the success or failure of learning is often determined by the suitability of the selected features. By comparison, subspace identification (SSID) methods are designed to select a feature set which preserves as much information as possible about state. In this paper we connect the two approaches, looking at the problem of reinforcement learning with a large set of features, each of which may only be marginally useful for value function approximation. We introduce a new algorithm for this situation, called Predictive State Temporal Difference (PSTD) learning. As in SSID for predictive state representations, PSTD finds a linear compression operator that projects a large set of features down to a small set that preserves the maximum amount of predictive information. As in RL, PSTD then uses a Bellman recursion to estimate a value function. We discuss the connection between PSTD and prior approaches in RL and SSID. We prove that PSTD is statistically consistent, perform several experiments that illustrate its properties, and demonstrate its potential on a difficult optimal stopping problem.

Motivation & Objective

• To address the challenge of feature selection in temporal difference learning, where high-dimensional or partially observed states lead to overfitting and poor generalization.
• To develop a method that automatically identifies a compact, predictive feature subspace from a large set of potentially weakly relevant features.
• To improve data efficiency and statistical consistency in value function approximation by leveraging insights from system identification and predictive state representations (PSRs).
• To demonstrate the practical superiority of PSTD over existing methods like LSTD and LARS-TD in real-world reinforcement learning problems, particularly in financial derivative pricing.

Proposed method

• PSTD uses subspace identification (SSID) to project a high-dimensional feature space into a lower-dimensional subspace that preserves maximum predictive information about future rewards.
• The algorithm applies a linear compression operator derived from time-adjacent feature samples to extract a minimal set of features that retain predictive power.
• It estimates the value function via a Bellman recursion on the compressed feature space, ensuring consistency with the underlying dynamical system.
• The method is theoretically grounded in predictive state representations (PSRs), with the compressed features asymptotically equivalent to the true PSR state under mild assumptions.
• PSTD is shown to be statistically consistent, with convergence dependent on the intrinsic dimension of the dynamical system rather than the size of the original state space.
• The approach is validated through both synthetic experiments and a real-world optimal stopping problem involving high-dimensional financial derivatives.

Experimental results

Research questions

• RQ1 Can a principled, data-driven method for feature compression improve the statistical consistency and data efficiency of temporal difference learning in high-dimensional or partially observable environments?
• RQ2 How does PSTD compare to LARS-TD and LSTD in terms of predictive accuracy when the feature set contains many marginally relevant features rather than a few highly relevant ones?
• RQ3 To what extent can PSTD’s compressed feature space serve as a consistent estimator of the true underlying system state, particularly in the context of PSRs?
• RQ4 Does the use of subspace identification for feature compression lead to measurable improvements in value function estimation and policy performance on real-world problems like financial derivative pricing?

Key findings

• PSTD outperformed LARS-TD by 1.75 percentage points and surpassed the best previously reported result by 1.24 percentage points in a high-dimensional optimal stopping problem for financial derivatives.
• The algorithm achieved consistent value function estimation by compressing 220 features down to 16, with the compressed features asymptotically equivalent to the true predictive state representation (PSR) state.
• PSTD demonstrated superior performance compared to LSTD with either the original 16 hand-selected features or the full 220-feature set, indicating that feature compression via predictive information maximization is more effective than manual feature selection or full feature use.
• The method’s statistical consistency was proven, with convergence depending on the intrinsic linear dimension of the dynamical system rather than the cardinality of the POMDP state space.
• In experiments, PSTD showed robustness and improved data efficiency, particularly when many features contribute only marginally to prediction, outperforming sparse selection methods like LARS-TD in such settings.
• The results suggest that PSTD enables significant arbitrage opportunities in derivative pricing, as improved value function estimation leads to more accurate security valuation.