[Paper Review] Consistency of Empirical Bayes And Kernel Flow For Hierarchical Parameter Estimation

Yifan Chen, Houman Owhadi+1 more|arXiv (Cornell University)|May 22, 2020

Gaussian Processes and Bayesian Inference41 references4 citations

TL;DR

This paper establishes the consistency of Empirical Bayes (EB) and Kernel Flow (KF) for hierarchical parameter estimation in Gaussian process regression, proving EB converges to the true regularity parameter $ s $, while KF converges to $ \frac{s - d/2}{2} $ in the large data limit. The key contribution is a theoretical analysis of implicit bias and robustness under model misspecification, with numerical experiments showing KF's superiority in misspecified settings despite EB's lower variance in well-specified cases.

ABSTRACT

Gaussian process regression has proven very powerful in statistics, machine learning and inverse problems. A crucial aspect of the success of this methodology, in a wide range of applications to complex and real-world problems, is hierarchical modeling and learning of hyperparameters. The purpose of this paper is to study two paradigms of learning hierarchical parameters: one is from the probabilistic Bayesian perspective, in particular, the empirical Bayes approach that has been largely used in Bayesian statistics; the other is from the deterministic and approximation theoretic view, and in particular the kernel flow algorithm that was proposed recently in the machine learning literature. Analysis of their consistency in the large data limit, as well as explicit identification of their implicit bias in parameter learning, are established in this paper for a Matérn-like model on the torus. A particular technical challenge we overcome is the learning of the regularity parameter in the Matérn-like field, for which consistency results have been very scarce in the spatial statistics literature. Moreover, we conduct extensive numerical experiments beyond the Matérn-like model, comparing the two algorithms further. These experiments demonstrate learning of other hierarchical parameters, such as amplitude and lengthscale; they also illustrate the setting of model misspecification in which the kernel flow approach could show superior performance to the more traditional empirical Bayes approach.

Motivation & Objective

• To analyze the consistency and implicit bias of Empirical Bayes (EB) and Kernel Flow (KF) in hierarchical parameter estimation for Gaussian processes.
• To establish theoretical convergence of both EB and KF estimators for the regularity parameter in a Matérn-like model on the torus.
• To compare robustness of EB and KF under model misspecification, particularly in recovering the regularity parameter and discontinuity positions.
• To extend the analysis beyond the Matérn-like model to amplitude, lengthscale, and variable-coefficient elliptic operators via numerical experiments.
• To provide a theoretical framework based on Fourier series for analyzing regularity parameter learning, with applications to well-specified and misspecified settings.

Proposed method

• Uses a Matérn-like kernel on the torus with a Fourier series characterization to analyze the regularity parameter $ s $.
• Applies empirical Bayes by maximizing the marginal likelihood under a hierarchical Gaussian process prior.
• Employs kernel flow as a deterministic, approximation-theoretic method minimizing the $ L^2 $ error between observed and predicted functions.
• Derives consistency results via large data limit analysis, proving convergence in probability for both EB and KF estimators.
• Introduces a Fourier-based toolkit to analyze the spectral properties of the kernel and the induced regularization.
• Conducts numerical experiments on well-specified and misspecified models to compare performance in recovering amplitude, lengthscale, and discontinuity parameters.

Experimental results

Research questions

• RQ1 Does the empirical Bayes estimator consistently recover the true regularity parameter $ s $ in the large data limit for a Matérn-like model?
• RQ2 Does the kernel flow estimator consistently recover a parameter related to the true regularity, and if so, what is its limiting value?
• RQ3 How do the implicit biases of EB and KF differ in terms of the regularity parameter, and what drives their distinct convergence behaviors?
• RQ4 How do EB and KF perform under model misspecification, particularly when the true process does not match the assumed kernel form?
• RQ5 Can the theoretical framework based on Fourier analysis be extended to recover multiple hyperparameters such as amplitude and lengthscale?

Key findings

• The empirical Bayes estimator converges in probability to the true regularity parameter $ s $ in the large data limit for the Matérn-like model.
• The kernel flow estimator converges in probability to $ \frac{s - d/2}{2} $, which corresponds to the minimal parameter achieving fast error rates in $ L^2 $-error.
• In well-specified models, EB exhibits lower variance in regularity estimation than KF, indicating better statistical efficiency when the prior is correctly specified.
• Under model misspecification—especially in discontinuity detection—kernel flow outperforms empirical Bayes, demonstrating greater robustness to incorrect modeling assumptions.
• The Fourier series toolkit enables rigorous analysis of the regularity parameter and proves the consistency of amplitude recovery under EB for the Matérn-like model.
• Numerical experiments confirm that both methods can recover amplitude, lengthscale, and discontinuity positions effectively in well-specified settings, but KF maintains performance under misspecification where EB fails.