[Paper Review] Structure Discovery in Nonparametric Regression through Compositional Kernel Search

David Duvenaud, James Robert Lloyd+3 more|arXiv (Cornell University)|Feb 20, 2013

Gaussian Processes and Bayesian Inference25 references274 citations

TL;DR

This paper proposes a compositional kernel search method for nonparametric regression that automatically discovers interpretable kernel structures by combining base kernels through sums and products. Using marginal likelihood as a criterion, the approach enables accurate long-range extrapolation in time series and outperforms standard kernels and combination methods on prediction tasks.

ABSTRACT

Despite its importance, choosing the structural form of the kernel in nonparametric regression remains a black art. We define a space of kernel structures which are built compositionally by adding and multiplying a small number of base kernels. We present a method for searching over this space of structures which mirrors the scientific discovery process. The learned structures can often decompose functions into interpretable components and enable long-range extrapolation on time-series datasets. Our structure search method outperforms many widely used kernels and kernel combination methods on a variety of prediction tasks.

Motivation & Objective

• To address the challenge of kernel selection in nonparametric regression, which often relies on expert intuition and trial-and-error.
• To automate the discovery of kernel structure rather than treating it as a fixed hyperparameter tuning problem.
• To enable interpretable decomposition of functions into meaningful components through compositional kernel construction.
• To improve generalization, especially long-range extrapolation in time-series data, by learning structurally meaningful kernels.
• To replace ad-hoc kernel engineering with a principled, search-based method grounded in Bayesian model selection.

Proposed method

• The method defines a space of kernel structures composed via addition and multiplication of a small set of base kernels (e.g., squared exponential, periodic, linear).
• It formulates kernel structure discovery as a discrete search problem over this compositional space, using marginal likelihood as the objective function.
• The search employs techniques inspired by equation discovery and unsupervised learning to explore complex kernel combinations efficiently.
• Kernels over multidimensional inputs are constructed by combining one-dimensional base kernels per input dimension using sum and product operations.
• The approach is applied within a Gaussian process framework, where the kernel defines the prior covariance and thus controls inductive bias.
• The method is evaluated using marginal likelihood to guide structure selection, enabling automatic discovery of meaningful functional decompositions.

Experimental results

Research questions

• RQ1 Can a systematic, automated method discover kernel structures that yield better generalization than hand-designed or standard kernel combinations?
• RQ2 Can the discovered kernel structures decompose complex functions into interpretable components, such as trend, periodicity, or local variation?
• RQ3 Does the compositional kernel search method enable accurate long-range extrapolation in time-series regression tasks?
• RQ4 How does the performance of the learned kernel structures compare to widely used kernel families and combination techniques?
• RQ5 Can the method recover known kernel forms from synthetic data, demonstrating its reliability and expressiveness?

Key findings

• The proposed method successfully recovers known kernel structures from synthetic data, validating its ability to discover correct functional forms.
• On real-world time-series datasets, the learned kernels enable accurate long-range extrapolation, outperforming standard kernels and combination methods.
• The discovered kernel structures often decompose functions into interpretable components, such as trend and periodicity, enhancing model interpretability.
• The method outperforms a variety of widely used kernel classes and kernel combination techniques on multiple supervised prediction tasks.
• Marginal likelihood-based search effectively guides the discovery of complex, compositionally structured kernels without requiring manual specification of kernel form.
• The approach demonstrates that automated kernel structure discovery can replace expert-driven kernel engineering with a transparent, science-based alternative.