[Paper Review] On Asymptotic Properties of Hyperparameter Estimators for Kernel-based Regularization Methods

Biqiang Mu, Tianshi Chen+1 more|arXiv (Cornell University)|Jul 3, 2017

Statistical Methods and Inference13 references3 citations

TL;DR

This paper analyzes the asymptotic properties of hyperparameter estimators in kernel-based regularization methods, focusing on empirical Bayes (EB) and two Stein's unbiased risk estimators (SURE). It shows that while SURE estimators asymptotically minimize mean square error and are thus optimal, the widely used EB estimator converges to a different, suboptimal target. Surprisingly, SUREs converge more slowly than EB, which is independent of the convergence rate of $\Phi^T\Phi/N$.

ABSTRACT

The kernel-based regularization method has two core issues: kernel design and hyperparameter estimation. In this paper, we focus on the second issue and study the properties of several hyperparameter estimators including the empirical Bayes (EB) estimator, two Stein's unbiased risk estimators (SURE) and their corresponding Oracle counterparts, with an emphasis on the asymptotic properties of these hyperparameter estimators. To this goal, we first derive and then rewrite the first order optimality conditions of these hyperparameter estimators, leading to several insights on these hyperparameter estimators. Then we show that as the number of data goes to infinity, the two SUREs converge to the best hyperparameter minimizing the corresponding mean square error, respectively, while the more widely used EB estimator converges to another best hyperparameter minimizing the expectation of the EB estimation criterion. This indicates that the two SUREs are asymptotically optimal but the EB estimator is not. Surprisingly, the convergence rate of two SUREs is slower than that of the EB estimator, and moreover, unlike the two SUREs, the EB estimator is independent of the convergence rate of $Φ^TΦ/N$ to its limit, where $Φ$ is the regression matrix and $N$ is the number of data. A Monte Carlo simulation is provided to demonstrate the theoretical results.

Motivation & Objective

• To understand the asymptotic behavior of hyperparameter estimators in kernel-based regularization methods.
• To compare the convergence targets and rates of empirical Bayes (EB) and two Stein’s unbiased risk estimators (SURE).
• To clarify whether EB or SURE estimators are asymptotically optimal in minimizing prediction error.

Proposed method

• Derive first-order optimality conditions for EB, SURE, and their Oracle counterparts.
• Rewrite optimality conditions in a common form to expose structural relationships between estimators.
• Analyze the limiting behavior of estimators as data size $N \to \infty$, focusing on convergence targets and rates.
• Use regularized least squares for FIR model estimation as a concrete framework.
• Conduct Monte Carlo simulations to validate theoretical findings.
• Leverage matrix calculus and asymptotic analysis under the assumption that $\Phi^T\Phi/N$ converges to a limit.

Experimental results

Research questions

• RQ1 As $N \to \infty$, what is the limiting value of the hyperparameter estimate for the EB estimator?
• RQ2 As $N \to \infty$, what is the limiting value of the hyperparameter estimate for the two SURE estimators?
• RQ3 Which estimator is asymptotically optimal in minimizing the mean square error?
• RQ4 How do the convergence rates of EB and SURE estimators compare?
• RQ5 Does the convergence rate of the EB estimator depend on the rate at which $\Phi^T\Phi/N$ converges to its limit?

Key findings

• As $N \to \infty$, the two SURE estimators converge to the hyperparameter that minimizes the corresponding mean square error, indicating asymptotic optimality.
• The empirical Bayes (EB) estimator converges to a different hyperparameter that minimizes the expectation of the EB estimation criterion, not the MSE, meaning EB is not asymptotically optimal.
• The convergence rate of the two SURE estimators is slower than that of the EB estimator.
• The EB estimator’s convergence is independent of the rate at which $\Phi^T\Phi/N$ converges to its limit, unlike the SURE estimators.
• Monte Carlo simulations confirm the theoretical convergence behavior and rates of the estimators.