[Paper Review] Asymptotic Confidence Sets for General Nonparametric Regression and Classification by Regularized Kernel Methods

Robert Hable|ERef Bayreuth (University of Bayreuth)|Mar 20, 2012

Numerical methods in inverse problems19 references3 citations

TL;DR

This paper establishes asymptotically valid confidence sets for general nonparametric functionals derived from regularized kernel methods, such as support vector machines and least-squares support vector regression. By proving the asymptotic normality of the estimator and deriving a strongly consistent estimator for the limiting covariance matrix, the method enables valid inference for functionals like pointwise values, gradients, integrals, and norms of the regularized solution.

ABSTRACT

Regularized kernel methods such as, e.g., support vector machines and least-squares support vector regression constitute an important class of standard learning algorithms in machine learning. Theoretical investigations concerning asymptotic properties have manly focused on rates of convergence during the last years but there are only very few and limited (asymptotic) results on statistical inference so far. As this is a serious limitation for their use in mathematical statistics, the goal of the article is to fill this gap. Based on asymptotic normality of many of these methods, the article derives a strongly consistent estimator for the unknown covariance matrix of the limiting normal distribution. In this way, we obtain asymptotically correct confidence sets for $ψ(f_{P,λ_0})$ where $f_{P,λ_0}$ denotes the minimizer of the regularized risk in the reproducing kernel Hilbert space $H$ and $ψ:H ightarrow\mathds{R}^m$ is any Hadamard-differentiable functional. Applications include (multivariate) pointwise confidence sets for values of $f_{P,λ_0}$ and confidence sets for gradients, integrals, and norms.

Motivation & Objective

• To address the lack of statistical inference tools for regularized kernel methods in mathematical statistics.
• To develop asymptotically valid confidence sets for functionals of the minimizer of the regularized risk in a reproducing kernel Hilbert space.
• To provide a consistent estimator for the asymptotic covariance matrix of the limiting normal distribution of the estimator.
• To enable inference on a broad class of functionals, including pointwise values, gradients, integrals, and norms of the learned function.

Proposed method

• Leverages asymptotic normality of regularized kernel estimators established in prior work (e.g., [10, Theorem 3.1]) to derive weak convergence of the scaled estimator to a Gaussian process.
• Applies functional delta-method principles to transform the weak convergence into a normal limit distribution for any Hadamard-differentiable functional ψ:H→ℝ^m.
• Derives a consistent estimator for the asymptotic covariance matrix Σ_P by solving a linear system involving the kernel matrix and second-order derivatives of the loss function.
• Uses a dual representation of the solution in the representer theorem framework, expressing the solution as a linear combination of kernel functions evaluated at training points.
• Constructs the covariance estimator via a matrix inversion involving the kernel matrix, the Hessian of the loss, and a basis for the span of the training features.
• Employs a projection onto a maximal linearly independent subset of the feature vectors to ensure numerical stability and existence of the inverse in the covariance estimation procedure.

Experimental results

Research questions

• RQ1 Can asymptotic confidence sets be constructed for general functionals of the regularized kernel estimator in a nonparametric setting?
• RQ2 Is it possible to derive a consistent estimator for the asymptotic covariance matrix of the limiting normal distribution of the regularized kernel estimator?
• RQ3 How can confidence sets be constructed for functionals such as pointwise values, gradients, integrals, and norms of the learned function?
• RQ4 What conditions ensure the consistency of the covariance estimator under general regularized kernel methods?

Key findings

• A strongly consistent estimator for the asymptotic covariance matrix of the regularized kernel estimator is derived, enabling valid asymptotic inference.
• The method supports confidence sets for a wide class of functionals, including pointwise values, gradients, integrals, and norms of the minimizer f_{P,λ₀}.
• The covariance estimator is constructed via a dual formulation involving the kernel matrix, the Hessian of the loss, and a basis of the training feature space.
• The consistency of the covariance estimator is established under the assumption that the regularized kernel estimator is asymptotically normal and the functional ψ is Hadamard-differentiable.
• The approach is applicable to standard regularized kernel methods such as support vector machines and least-squares support vector regression.
• The method enables inference on the true regularized solution f_{P,λ₀}, though not on the unregularized minimizer due to lack of uniform convergence results in the nonparametric setting.