[Paper Review] Kernel regression, minimax rates and effective dimensionality: beyond the regular case

Gilles Blanchard, Nicole Mücke|arXiv (Cornell University)|Nov 12, 2016

Numerical methods in inverse problems19 references4 citations

TL;DR

This paper establishes minimax convergence rates for kernel regression under weak eigenvalue decay assumptions on the kernel covariance operator, demonstrating that optimal rates are achievable even when the spectrum does not decay polynomially. By introducing a novel effective dimensionality measure based on eigenvalue decay, the authors derive sharp lower bounds that generalize classical minimax rates to distribution-free settings.

ABSTRACT

We investigate if kernel regularization methods can achieve minimax convergence rates over a source condition regularity assumption for the target function. These questions have been considered in past literature, but only under specific assumptions about the decay, typically polynomial, of the spectrum of the the kernel mapping covariance operator. In the perspective of distribution-free results, we investigate this issue under much weaker assumption on the eigenvalue decay, allowing for more complex behavior that can reflect different structure of the data at different scales.

Motivation & Objective

• To determine whether kernel regularization methods can achieve minimax convergence rates without assuming polynomial eigenvalue decay of the kernel covariance operator.
• To extend minimax theory in kernel regression to distribution-free settings where the sampling measure may not be dominated by a reference measure.
• To characterize the effective dimensionality of the problem through a new notion based on eigenvalue decay, rather than intrinsic dimensionality.
• To derive sharp lower bounds on estimation error that reflect the true complexity of the learning problem under general spectral assumptions.
• To bridge the gap between classical nonparametric minimax theory and modern kernel methods in high-dimensional or non-standard spaces.

Proposed method

• Proposes a new class of regularity classes for the target function based on the eigen-decomposition of the kernel covariance operator, defined via $ \Omega(\nu, r, R) = \{ f \in \mathcal{H} : \sum_i \mu_{\nu,i}^r f_i^2 \leq R^2 \} $.
• Introduces a generalized notion of effective dimensionality through the function $ \mathcal{F}(x) = \# \{ i : \mu_{\nu,i} \geq x \} $, which captures non-polynomial eigenvalue decay.
• Employs Le Cam's method and the use of a finite set of well-separated distributions to derive minimax lower bounds via testing arguments.
• Applies a construction of $ N_{\varepsilon} $ functions in the regularity class with controlled $ \mathcal{H} $-norm and separation in the $ B^s $-norm.
• Uses Kullback-Leibler divergence bounds between data distributions to control the information-theoretic cost of distinguishing hypotheses.
• Applies a variant of the Fano inequality with a logarithmic term in the number of hypotheses to derive a non-asymptotic lower bound on estimation error.

Experimental results

Research questions

• RQ1 Can kernel ridge regression achieve minimax rates under weak eigenvalue decay assumptions, beyond the standard polynomial decay case?
• RQ2 How does the effective dimensionality of the learning problem depend on the spectral structure of the kernel covariance operator in non-regular settings?
• RQ3 What is the optimal rate of convergence for kernel regression when the eigenvalues decay faster than polynomially, and how does it compare to classical Sobolev-type rates?
• RQ4 Can distribution-free minimax lower bounds be derived without assuming the sampling measure is dominated by a reference measure?
• RQ5 To what extent does the geometry of the data, as encoded in the kernel's eigenvalues, determine the fundamental limits of kernel-based learning?

Key findings

• The paper establishes a minimax lower bound of order $ \Omega\left( \left( \frac{\sigma^2}{n} \right)^{\frac{r+s}{r+s+1}} \right) $ for the $ B^s $-norm error, which matches known upper bounds under mild conditions.
• The effective dimensionality is characterized by the function $ \mathcal{F}(x) = \# \{ i : \mu_{\nu,i} \geq x \} $, which generalizes the notion of intrinsic dimensionality to non-polynomial decay.
• For any $ \varepsilon > 0 $ sufficiently small, there exist $ N_{\varepsilon} \to \infty $ as $ \varepsilon \to 0 $, such that $ \log(N_{\varepsilon}) \geq \frac{1}{36} \mathcal{F}(2^{\nu_*} (\varepsilon/R)^{1/(r+s)}) $, showing the richness of the hypothesis class.
• The Kullback-Leibler divergence between data distributions is bounded by $ \mathcal{K}(\mathbb{P}_i, \mathbb{P}_j) \leq C_{\nu_*,s} R^2 \sigma^{-2} (\varepsilon/R)^{(2r+1)/(r+s)} $, which controls the distinguishability of hypotheses.
• The lower bound holds under the assumption $ \text{Eigen}^>(\nu_*) $, which allows for arbitrary spectral decay as long as the decay is regular enough on a logarithmic scale.
• The derived minimax rate is sharp and matches known upper bounds, showing that kernel methods can achieve optimal rates even when eigenvalues decay faster than polynomially.