[Paper Review] Fast Gaussian Process Based Gradient Matching for Parameter Identification in Systems of Nonlinear ODEs

Philippe Wenk, Alkis Gotovos+4 more|arXiv (Cornell University)|Apr 12, 2018

Gaussian Processes and Bayesian Inference26 references12 citations

TL;DR

This paper proposes Fast Gaussian Process-based Gradient Matching (FGPGM), a novel Bayesian inference method for parameter identification in nonlinear ODE systems. By replacing the theoretically flawed product of experts heuristic with a sound probabilistic formulation and integrating variational inference with sequential hyperparameter optimization, FGPGM achieves 35% faster runtime, up to 62% lower RMSE, and reduced smoothing bias compared to state-of-the-art methods, while enabling simpler, more robust MCMC sampling.

ABSTRACT

Parameter identification and comparison of dynamical systems is a challenging task in many fields. Bayesian approaches based on Gaussian process regression over time-series data have been successfully applied to infer the parameters of a dynamical system without explicitly solving it. While the benefits in computational cost are well established, a rigorous mathematical framework has been missing. We offer a novel interpretation which leads to a better understanding and improvements in state-of-the-art performance in terms of accuracy for nonlinear dynamical systems.

Motivation & Objective

• To address theoretical inconsistencies in the widely used product of experts (PoE) heuristic for Gaussian process-based gradient matching in nonlinear ODEs.
• To explain the surprising performance gains of variational inference over MCMC in existing methods.
• To develop a more accurate, robust, and computationally efficient algorithm for parameter identification in nonlinear dynamical systems.
• To reduce the smoothing bias inherent in GP-based gradient matching by improving the modeling framework.

Proposed method

• Replaces the heuristic product of experts (PoE) approach with a principled probabilistic formulation that properly combines GP-derived derivatives and ODE-predicted derivatives.
• Introduces a joint inference scheme that sequentially optimizes hyperparameters and samples from the posterior using a one-chain Metropolis-Hastings algorithm.
• Employs variational inference to approximate the posterior over states and parameters, enabling faster computation than MCMC-based alternatives.
• Uses a Matérn-5/2 kernel to better capture spiky, nonlinear dynamics in systems like the FHN neuron model.
• Applies a hierarchical prior structure that avoids the need for complex, hard-to-justify hyperpriors used in previous methods.
• Employs a sequential fitting procedure for hyperparameters, which stabilizes the inference process and avoids pathological posterior density behavior.

Experimental results

Research questions

• RQ1 Why does the product of experts (PoE) heuristic lead to theoretical inconsistencies in GP-based gradient matching for ODEs?
• RQ2 What explains the superior empirical performance of variational inference over MCMC in existing GP-based ODE parameter estimation?
• RQ3 Can a theoretically sound, computationally efficient, and robust alternative to PoE be developed for nonlinear ODE systems?
• RQ4 How does the proposed method reduce the smoothing bias that distorts fast or spiky dynamics in ODEs?

Key findings

• FGPGM reduces run time by approximately 35% compared to the state-of-the-art AGM method, while maintaining or improving accuracy.
• In the high-noise Lotka-Volterra system, FGPGM reduces median state RMSE by 62% compared to AGM and 31% compared to MVGM.
• For the nonlinear Protein Transduction system, FGPGM reduces variance in state estimates R and Rpp by at least one order of magnitude and decreases bias by over 30%.
• FGPGM significantly reduces the smoothing bias, especially evident in spiky dynamics such as the FHN neuron model, with improved fidelity even at low observation counts (e.g., 10 observations).
• The method achieves higher accuracy and robustness than MCMC-based approaches, while using a simpler, one-chain Metropolis-Hastings sampler that is easier to tune.
• The proposed framework successfully avoids the need for complex, hard-to-justify hyperpriors and eliminates the pathological posterior density issues seen in previous multi-chain MCMC setups.