[Paper Review] Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constants

Liyao Gao, J. Nathan Kutz|arXiv (Cornell University)|Nov 19, 2022

Gaussian Processes and Bayesian Inference4 citations

TL;DR

This paper proposes Bayesian SINDy autoencoders that jointly discover latent coordinates and governing equations from low-data, high-noise spatio-temporal data using a hierarchical Spike-and-slab Gaussian-Laplace prior for sparse identification. The method enables computationally tractable Bayesian posterior sampling via Stochastic Gradient Langevin Dynamics (SGLD), achieving accurate physics discovery with uncertainty quantification—demonstrated by successfully recovering the pendulum equation and estimating gravity $g$ from just 14 seconds of real video data.

ABSTRACT

Recent progress in autoencoder-based sparse identification of nonlinear dynamics (SINDy) under $\ell_1$ constraints allows joint discoveries of governing equations and latent coordinate systems from spatio-temporal data, including simulated video frames. However, it is challenging for $\ell_1$-based sparse inference to perform correct identification for real data due to the noisy measurements and often limited sample sizes. To address the data-driven discovery of physics in the low-data and high-noise regimes, we propose Bayesian SINDy autoencoders, which incorporate a hierarchical Bayesian sparsifying prior: Spike-and-slab Gaussian Lasso. Bayesian SINDy autoencoder enables the joint discovery of governing equations and coordinate systems with a theoretically guaranteed uncertainty estimate. To resolve the challenging computational tractability of the Bayesian hierarchical setting, we adapt an adaptive empirical Bayesian method with Stochatic gradient Langevin dynamics (SGLD) which gives a computationally tractable way of Bayesian posterior sampling within our framework. Bayesian SINDy autoencoder achieves better physics discovery with lower data and fewer training epochs, along with valid uncertainty quantification suggested by the experimental studies. The Bayesian SINDy autoencoder can be applied to real video data, with accurate physics discovery which correctly identifies the governing equation and provides a close estimate for standard physics constants like gravity $g$, for example, in videos of a pendulum.

Motivation & Objective

• To address the challenge of physics discovery in low-data, high-noise regimes where traditional sparse regression fails due to instability and poor generalization.
• To integrate Bayesian sparse inference with autoencoders for joint discovery of latent coordinates and governing equations in dynamical systems.
• To provide theoretically grounded uncertainty quantification for discovered models, crucial for reliable data-driven decision-making.
• To overcome the computational intractability of hierarchical Bayesian models in large-scale or high-dimensional settings through adaptive empirical Bayes and SGLD sampling.
• To enable robust discovery from real-world video data—such as pendulum motion—where temporal derivatives are noisy and sample sizes are limited.

Proposed method

• The framework combines a deep autoencoder with a Bayesian SINDy module, where the SINDy layer applies a Spike-and-slab Gaussian-Laplace (SSGL) prior to enforce sparsity in the governing equation coefficients.
• The SSGL prior enables joint variable selection and uncertainty estimation by modeling each coefficient as a mixture of a point mass at zero (spike) and a Laplace distribution (slab).
• An adaptive empirical Bayes approach is used to approximate the posterior distribution over model parameters, avoiding the need for full MCMC sampling.
• Stochastic Gradient Langevin Dynamics (SGLD) is employed to perform Bayesian posterior sampling efficiently, enabling scalable training on real video data.
• The method jointly optimizes the autoencoder reconstruction loss, SINDy prediction loss, and Bayesian evidence, with uncertainty propagated through posterior samples.
• The architecture is trained end-to-end with a differentiable SGLD-based posterior approximation, allowing gradient-based optimization while preserving uncertainty estimates.

Experimental results

Research questions

• RQ1 Can Bayesian SINDy autoencoders outperform standard SINDy autoencoders in discovering governing equations and latent coordinates under low-data and high-noise conditions?
• RQ2 How well can the Bayesian framework quantify uncertainty in discovered models when the true dynamics are misspecified or data are noisy?
• RQ3 Does the use of a hierarchical Spike-and-slab prior improve model selection accuracy and robustness compared to standard L1-based sparsity in real video data?
• RQ4 Can the method successfully recover fundamental physical constants—such as gravity $g$—from minimal real-world video footage?
• RQ5 To what extent does uncertainty quantification from the Bayesian framework improve reliability and interpretability of discovered models?

Key findings

• The Bayesian SINDy autoencoder successfully discovered the pendulum’s governing equation $\ddot{z} = -g\sin(z)$ using only 14 seconds of real video data, with a reconstructed $g$ value close to the true physical constant.
• The method achieved a decoder loss of 0.779, a reconstruction loss of 0.364 for $\ddot{x}$, and 0.416 for $\ddot{z}$, with a fraction of unexplained variance of 0.012 in reconstruction.
• The fraction of unexplained variance in SINDy predictions was 0.210 for $\ddot{x}$ and 0.941 for $\ddot{z}$, indicating strong performance on the latent dynamics despite noise.
• The Bayesian framework provided valid uncertainty estimates, as visualized in posterior samples, even when the Laplace prior failed to identify the correct term $\sin(z)$.
• The Spike-and-slab prior outperformed the Laplace prior in both model discovery accuracy and visual reconstruction quality, with the latter identifying an incorrect term $z$ instead of $\sin(z)$.
• The method required only 1,500 epochs to train and achieved a 70% success rate in discovery across 10 trials, with the remaining 30% failing to identify active terms due to model misspecification.