[Paper Review] On the Multiple Descent of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels

Tengyuan Liang, Alexander Rakhlin+1 more|arXiv (Cornell University)|Aug 27, 2019

Stochastic Gradient Optimization TechniquesComputer Science59 citations

TL;DR

The paper analyzes the risk of minimum-norm interpolants in RKHS under various high-dimensional scaling regimes and proves a restricted lower isometry property for high-dimensional kernel matrices, revealing a non-monotone, multi-descent risk behavior as d scales with n.

ABSTRACT

We study the risk of minimum-norm interpolants of data in Reproducing Kernel Hilbert Spaces. Our upper bounds on the risk are of a multiple-descent shape for the various scalings of $d = n^{\\alpha}$, $\\alpha\\in(0,1)$, for the input dimension $d$ and sample size $n$. Empirical evidence supports our finding that minimum-norm interpolants in RKHS can exhibit this unusual non-monotonicity in sample size; furthermore, locations of the peaks in our experiments match our theoretical predictions. Since gradient flow on appropriately initialized wide neural networks converges to a minimum-norm interpolant with respect to a certain kernel, our analysis also yields novel estimation and generalization guarantees for these over-parametrized models. At the heart of our analysis is a study of spectral properties of the random kernel matrix restricted to a filtration of eigen-spaces of the population covariance operator, and may be of independent interest.

Motivation & Objective

• Motivate understanding of generalization and consistency for minimum-norm interpolants (kernel ridgeless regression) in RKHS.
• Characterize risk behavior under high-dimensional scaling d ~ n^α, α in (0,1).
• Reveal spectral properties of randomized kernel matrices via a restricted lower isometry analysis.
• Connect results to gradient-flow trained over-parameterized models and NTK-like kernels.

Proposed method

• Study minimum-norm interpolants f̂ in RKHS defined by kernel k(x,z)=h(x^Tz/d) with h smooth and nonnegative Taylor coefficients.
• Analyze the variance and bias contributions of the interpolant using a closed-form f̂(x)=k(x,X)^T K^{-1} Y and a bias-variance decomposition conditional on X.
• Establish a restricted lower isometry property for the empirical kernel matrix on a filtration of eigen-spaces of the population covariance operator.
• Use a Gram-Schmidt orthogonalization on polynomial features to control the covariance structure of monomials, enabling spectral lower bounds.
• Apply small-ball probability techniques to bound the smallest eigenvalues of the sample covariance in high dimensions.
• Extend results to neural-network motivated kernels, including a Neural-Tangent-Type kernel, and derive generalization bounds.

Experimental results

Research questions

• RQ1 How does the risk of minimum-norm kernel interpolants behave under regimes where dimension scales as d ~ n^α with α in (0,1)?
• RQ2 Can a Restricted Lower Isometry Property (RLIP) be established for the high-dimensional kernel matrix, and how does it influence variance and bias bounds?
• RQ3 Do these RKHS results extend to neural-network trained regimes via NTK-type kernels, providing guarantees for over-parameterized models?
• RQ4 Where are the regimes of peak risk (multi-descent) located, and how do they relate to spectral properties of the population covariance?
• RQ5 Under noiseless vs noisy settings, how does the interpolant’s generalization performance change?

Key findings

• The risk upper bound of minimum-norm interpolants in RKHS exhibits a multiple-descent shape as d scales with n across regimes d ≈ n^α, α ∈ (0,1).
• For each integer ι≥1 and α ∈ [1/(ι+1), 1/ι), the risk curve has a valley with fast rates around d ≈ n^{1/(ι+1/2)} and shows peaks at these predicted scales.
• Empirical kernel matrices, when restricted to a filtration of population-eigen spaces, satisfy a restricted lower isometry property, enabling sharp variance and bias control.
• The variance bound includes a term proportional to d^ι/n plus n/d^{ι+1}, and the results cover non-polynomial, as well as polynomial, kernel cases depending on h’s Taylor coefficients.
• Bias can be controlled in terms of the variance term, with bounds that hold under assumptions on the target function’s representation via the kernel and bounded kernel values.
• Corollaries extend the main results to Neural-Tangent-Type kernels, implying estimation and generalization guarantees for wide neural networks converging to minimum-norm interpolants.