[Paper Review] Gaussian process modelling of multiple short time series

Hande Topa, Antti Honkela|arXiv (Cornell University)|Oct 9, 2012

Gaussian Processes and Bayesian Inference17 references3 citations

TL;DR

This paper proposes constrained Gaussian process (GP) modeling for multiple short time series, addressing over-fitting and under-fitting in high-throughput biological data. By rigorously deriving a frequency-based bound on the GP length-scale and using informative priors on observation noise, the method enables reliable automatic fitting of thousands of short time series without manual intervention, validated on synthetic and gene expression data with significant reduction in problematic fits.

ABSTRACT

We present techniques for effective Gaussian process (GP) modelling of multiple short time series. These problems are common when applying GP models independently to each gene in a gene expression time series data set. Such sets typically contain very few time points. Naive application of common GP modelling techniques can lead to severe over-fitting or under-fitting in a significant fraction of the fitted models, depending on the details of the data set. We propose avoiding over-fitting by constraining the GP length-scale to values that focus most of the energy spectrum to frequencies below the Nyquist frequency corresponding to the sampling frequency in the data set. Under-fitting can be avoided by more informative priors on observation noise. Combining these methods allows applying GP methods reliably automatically to large numbers of independent instances of short time series. This is illustrated with experiments with both synthetic data and real gene expression data.

Motivation & Objective

• To address over-fitting and under-fitting in Gaussian process modeling of large numbers of short time series, common in gene expression data.
• To provide a rigorous theoretical foundation for length-scale bounds previously used heuristically in biological GP applications.
• To enable fully automatic, reliable GP fitting across thousands of independent short time series without manual model tuning.
• To improve the accuracy of downstream analyses such as gene regulatory network inference by stabilizing GP model fits.
• To support the transition to more complex, flexible GP models in systems biology by preventing unexpected over-fitting failure modes.

Proposed method

• Derives a rigorous bound on the GP length-scale based on the Nyquist frequency of the sampling rate, ensuring most energy is concentrated in reconstructable frequency bands.
• Applies a hard upper bound on the length-scale parameter ℓ to prevent over-fitting from excessive functional flexibility.
• Uses more informative priors on observation noise σ²ₙ to reduce under-fitting in low-signal regimes.
• Combines bounded ℓ with fixed σ²ₙ from pre-processing to stabilize model fitting across diverse data instances.
• Employs maximum likelihood estimation with constrained hyperparameters to ensure robustness in high-throughput settings.
• Validates the approach using both synthetic data and real gene expression time series from systems biology.

Experimental results

Research questions

• RQ1 What causes over-fitting in Gaussian process models fitted to short time series with few observations?
• RQ2 How can a theoretically justified length-scale bound be derived to prevent over-fitting in GP models for short time series?
• RQ3 What role do informative priors on observation noise play in mitigating under-fitting in sparse data regimes?
• RQ4 How do length-scale constraints and noise priors jointly improve the reliability of large-scale GP fitting across thousands of independent time series?
• RQ5 To what extent do these constraints reduce problematic model fits in real biological data, such as gene expression time series?

Key findings

• The proposed length-scale bound significantly reduces over-fitting, with 7.3% of models showing ℓ < aₗ without constraints, dropping to 0% when bounded.
• Observation noise priors reduce under-fitting, decreasing the proportion of models with σ²ₙ < 0.01 from 8.3% to 6.7% when noise is fixed.
• In single-target ODE models, the combination of bounded ℓ and fixed σ²ₙ corrected severe over-fitting in weakly expressed regulators like TIN.
• The method enables reliable automatic fitting of GP models across 6,795 genes, reducing manual intervention in large-scale analyses.
• The likelihood surface analysis confirms that small length-scales are not artifacts of point estimation but reflect genuine over-fitting tendencies in sparse data.
• The approach is effective even in complex models like non-cascaded ODEs with multiple targets, where over-fitting rates remain low when constraints are applied.