[Paper Review] Scaling up the Automatic Statistician: Scalable Structure Discovery using Gaussian Processes

Hyunjik Kim, Yee Whye Teh|arXiv (Cornell University)|Jun 8, 2017

Gaussian Processes and Bayesian Inference31 references18 citations

TL;DR

This paper proposes Scalable Kernel Composition (SKC), a scalable extension of the Automatic Statistician that enables interpretable Gaussian Process (GP) model discovery for larger datasets by introducing a cheap, tight upper bound on the GP marginal likelihood. By sandwiching the true marginal likelihood between this upper bound and the variational lower bound, SKC enables efficient, semi-greedy kernel search that successfully identifies complex structures like multi-scale periodicities and trends in data with tens of thousands of points—scaling from O(N³) to O(N²) complexity.

ABSTRACT

Automating statistical modelling is a challenging problem in artificial intelligence. The Automatic Statistician takes a first step in this direction, by employing a kernel search algorithm with Gaussian Processes (GP) to provide interpretable statistical models for regression problems. However this does not scale due to its $O(N^3)$ running time for the model selection. We propose Scalable Kernel Composition (SKC), a scalable kernel search algorithm that extends the Automatic Statistician to bigger data sets. In doing so, we derive a cheap upper bound on the GP marginal likelihood that sandwiches the marginal likelihood with the variational lower bound . We show that the upper bound is significantly tighter than the lower bound and thus useful for model selection.

Motivation & Objective

• To address the scalability limitation of the Automatic Statistician, which is constrained to small datasets due to O(N³) GP inference complexity.
• To enable automated, interpretable statistical modelling for medium- and large-scale datasets by extending Compositional Kernel Search (CKS) to larger data.
• To develop a scalable model selection framework that maintains interpretability while discovering complex structures such as multi-periodic trends and local variations.
• To provide a reliable, tight upper bound on the GP marginal likelihood that improves model selection over variational lower bounds alone.

Proposed method

• Introduce a novel, computationally cheap upper bound on the GP marginal likelihood that tightly sandwiches the true marginal likelihood between it and the variational lower bound.
• Use the upper bound in a semi-greedy kernel search strategy to guide exploration of the compositional kernel space, improving robustness to local optima.
• Apply the upper bound as a proxy for model selection in place of the exact marginal likelihood, enabling scalable kernel composition without full GP inference.
• Leverage variational inference with inducing points to approximate the GP marginal likelihood, reducing computational cost from O(N³) to O(N²).
• Use the upper bound as a tie-breaker for kernels with overlapping lower bound intervals, enhancing stability and accuracy in model selection.
• Apply the method to time series data using base kernels (SE, LIN, PER) to compose interpretable, structure-aware GPs.

Experimental results

Research questions

• RQ1 Can we scale the Automatic Statistician to datasets with tens of thousands of points while preserving model interpretability?
• RQ2 Is a cheap, tight upper bound on the GP marginal likelihood effective for guiding kernel search in large-scale settings?
• RQ3 How does the proposed upper bound compare in tightness and reliability to the variational lower bound for model selection?
• RQ4 Can SKC discover complex, multi-scale structures such as doubly periodic patterns and local trends in large time series?
• RQ5 Does the use of the upper bound enable deeper kernel search (e.g., beyond depth 4) compared to using only the lower bound or random subsampling?

Key findings

• SKC successfully discovers a kernel with SE × PER₁ × LIN × PER₂ × PER₃ structure in tidal data with N = 31,957 points, capturing both 12-hour and bi-weekly periodicities.
• The upper bound on the marginal likelihood is significantly tighter than the variational lower bound, enabling more reliable and stable model selection.
• Kernel search using only the lower bound fails to proceed past depth 3 due to poor signal-to-noise in the bound, while SKC successfully explores deeper structures.
• Random subsampling of data fails to preserve global structures like long-periodic trends, even at sizes up to 2,560 points, whereas SKC with m = 640 inducing points captures them accurately.
• The LIN kernel has a negligible effect (slope = -5×10⁻⁶), and PER₂ has minimal impact due to high length scale and low magnitude, confirming the model's interpretability.
• SKC achieves scalable, interpretable structure discovery in large datasets where CKS and subsampling-based approaches fail, demonstrating O(N²) runtime scaling.