[Paper Review] Hilbert Space Embeddings of Predictive State Representations

Byron Boots, Geoffrey J. Gordon+1 more|arXiv (Cornell University)|Sep 26, 2013

Gaussian Processes and Bayesian Inference25 references36 citations

TL;DR

This paper introduces Hilbert Space Embeddings of Predictive State Representations (HSE-PSRs), a nonparametric method that generalizes discrete PSRs to continuous actions and observations by representing state as conditional distribution embeddings in a Reproducing Kernel Hilbert Space (RKHS). Using kernel Bayes' rule and moment-based learning, HSE-PSRs achieve statistically consistent estimation without density estimation, outperforming competing system identification methods on continuous control benchmarks with lower prediction error.

ABSTRACT

Predictive State Representations (PSRs) are an expressive class of models for controlled stochastic processes. PSRs represent state as a set of predictions of future observable events. Because PSRs are defined entirely in terms of observable data, statistically consistent estimates of PSR parameters can be learned efficiently by manipulating moments of observed training data. Most learning algorithms for PSRs have assumed that actions and observations are finite with low cardinality. In this paper, we generalize PSRs to infinite sets of observations and actions, using the recent concept of Hilbert space embeddings of distributions. The essence is to represent the state as a nonparametric conditional embedding operator in a Reproducing Kernel Hilbert Space (RKHS) and leverage recent work in kernel methods to estimate, predict, and update the representation. We show that these Hilbert space embeddings of PSRs are able to gracefully handle continuous actions and observations, and that our learned models outperform competing system identification algorithms on several prediction benchmarks.

Motivation & Objective

• To extend Predictive State Representations (PSRs) to continuous and high-cardinality action and observation spaces.
• To overcome limitations of prior continuous PSR methods that rely on parametric families or density estimation.
• To develop a statistically consistent learning algorithm for PSRs using kernel methods and Hilbert space embeddings.
• To enable accurate prediction in partially observable, nonlinear dynamical systems using only unlabeled sequences of actions and observations.
• To demonstrate superior performance over existing system identification methods on continuous control tasks.

Proposed method

• Represents state as nonparametric conditional embedding operators in a Reproducing Kernel Hilbert Space (RKHS), using kernelized representations of test and history distributions.
• Employs kernel Bayes' rule for filtering and state update, enabling Bayesian inference in RKHS without explicit density estimation.
• Leverages the kernel trick to implicitly represent state and operators, maintaining a state vector size proportional to training data.
• Uses moment-based learning via singular value decomposition on joint probability matrices of tests and histories, ensuring statistical consistency.
• Applies Gaussian RBF kernels to model continuous observations and actions, enabling nonparametric modeling of complex dynamics.
• Maintains linear prediction functions in the RKHS, allowing efficient filtering and prediction using linear algebra.

Experimental results

Research questions

• RQ1 Can PSRs be generalized to continuous and high-cardinality action and observation spaces while preserving statistical consistency?
• RQ2 Can Hilbert space embeddings of distributions enable accurate, nonparametric modeling of predictive state dynamics without assuming parametric families?
• RQ3 How does the performance of HSE-PSRs compare to existing system identification methods in continuous control tasks?
• RQ4 Can the proposed method learn from unlabeled sequences of actions and observations without requiring labeled latent states?
• RQ5 Does the kernel-based learning algorithm achieve better prediction accuracy than EM-based or parametric alternatives?

Key findings

• HSE-PSRs outperform K-LDS, LDS, and IO-HMM baselines on the cart-pole benchmark, achieving the lowest mean squared error (MSE) across all prediction horizons.
• On the slot car dataset, HSE-PSR achieved the lowest RMSE in predicting IMU readings, significantly outperforming all competing models, including stabilized LDS and IO-HMM.
• In the robot end-effector prediction task, HSE-PSR yielded significantly lower mean prediction error than K-LDS and stabilized LDS, which produced unstable or inaccurate models.
• The HSE-PSR model achieved consistent performance across all benchmarks without requiring density estimation or numerical integration.
• The method demonstrated robustness to non-stationary dynamics and complex nonlinearities, particularly in high-dimensional continuous observation spaces.
• The algorithm maintained statistical consistency and scalability, with model size growing linearly with training data, enabling practical deployment.