[Paper Review] Bayesian Alignments of Warped Multi-Output Gaussian Processes

Markus Kaiser, Clemens Otte+2 more|arXiv (Cornell University)|Oct 8, 2017

Gaussian Processes and Bayesian Inference18 references3 citations

TL;DR

This paper proposes a Bayesian framework for warped multi-output Gaussian processes (AMO-GP) that models nonlinear temporal alignments and shared latent structures across multiple time series. By introducing a shared convolutional process layer with variational inference and closed-form expectations for kernel statistics, the model enables flexible, principled uncertainty propagation and outperforms standard GPs in capturing cross-output dependencies and nonlinear warping.

ABSTRACT

We propose a novel Bayesian approach to modelling nonlinear alignments of time series based on latent shared information. We apply the method to the real-world problem of finding common structure in the sensor data of wind turbines introduced by the underlying latent and turbulent wind field. The proposed model allows for both arbitrary alignments of the inputs and non-parametric output warpings to transform the observations. This gives rise to multiple deep Gaussian process models connected via latent generating processes. We present an efficient variational approximation based on nested variational compression and show how the model can be used to extract shared information between dependent time series, recovering an interpretable functional decomposition of the learning problem. We show results for an artificial data set and real-world data of two wind turbines.

Motivation & Objective

• To model complex, nonlinear temporal alignments between multiple time series where standard multi-output GPs fail due to rigid alignment assumptions.
• To enable information sharing across multiple deep Gaussian processes through a shared convolutional process layer that captures relative time offsets and nonlinear warping.
• To develop a variational inference scheme with closed-form expectations for kernel statistics, allowing efficient optimization and uncertainty quantification.
• To recover latent shared dynamics and individual output-specific warping functions from data, particularly in scenarios with misaligned or nonlinearly transformed observations.
• To improve modeling flexibility and accuracy over standard multi-output and deep Gaussian processes by incorporating principled Bayesian treatment of alignments and warping.

Proposed method

• Introduces a shared convolutional process (CP) layer that acts as a common interface for multiple deep Gaussian processes, enabling information sharing across outputs.
• Employs a Bayesian treatment of latent time alignments via variational inference, with a variational distribution q(a) ∼ N(μa, Σa) over alignment variables.
• Derives closed-form expressions for key Ψ-statistics (Ψf = E[Kfu], Φf = E[KufKfu], ψf = E[tr(Kff)]) over kernel matrices under the GP prior and variational approximation.
• Utilizes a warping layer g_d for each output to model nonlinear time transformations, enabling flexible alignment beyond constant offsets.
• Adapts nested variational compression for multi-output deep GP inference, extending the approach to handle multiple GPs simultaneously with shared inducing variables.
• Employs stochastic optimization by exploiting factorization of expectations along data points, enabling efficient computation in O(NM²) time.

Experimental results

Research questions

• RQ1 Can a Bayesian framework effectively model nonlinear temporal alignments between multiple time series while preserving uncertainty in the alignment parameters?
• RQ2 How can shared latent structure be modeled across multiple outputs when the observations are misaligned or nonlinearly warped in time?
• RQ3 What closed-form solutions exist for kernel expectation statistics (Ψf, Φf, ψf) under a convolutional process with uncertain alignments?
• RQ4 To what extent does the proposed AMO-GP model improve predictive performance and structural recovery compared to standard multi-output and deep Gaussian processes?
• RQ5 Can the model recover both the latent shared dynamics and individual warping functions from data, particularly in the presence of nonlinear time distortions?

Key findings

• The AMO-GP model successfully recovers the true nonlinear alignment and warping functions in synthetic data, outperforming standard multi-output and deep GP models.
• The model captures cross-output dependencies that standard multi-output GPs fail to model due to their assumption of fixed time offsets.
• Closed-form solutions for Ψ-statistics (Ψf, Φf, ψf) are derived under the RBF and linear kernel assumptions, enabling efficient stochastic optimization.
• The shared convolutional process layer enables effective information sharing across multiple deep GPs, improving generalization and predictive performance.
• On a real-world wind turbine dataset, the AMO-GP successfully recovers the latent prevailing wind conditions and relative timing of wind fronts hitting the turbines.
• The model avoids reverting to the prior in regions of high uncertainty, unlike shallow and deep GP models, due to its ability to learn and propagate alignment uncertainty.