[Paper Review] On the Risk of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels.

Tengyuan Liang, Alexander Rakhlin+1 more|arXiv (Cornell University)|Aug 27, 2019

Stochastic Gradient Optimization Techniques19 citations

TL;DR

This paper analyzes the generalization risk of minimum-norm interpolants in Reproducing Kernel Hilbert Spaces (RKHS), showing that the risk exhibits a multiple-descent behavior as a function of sample size n and input dimension d = n^α for α ∈ (0,1). The analysis reveals non-monotonic risk curves with peaks matching theoretical predictions, and extends to over-parameterized neural networks via kernel equivalence.

ABSTRACT

We study the risk of minimum-norm interpolants of data in Reproducing Kernel Hilbert Spaces. Our upper bounds on the risk are of a multiple-descent shape for the various scalings of $d = n^{\alpha}$, $\alpha\in(0,1)$, for the input dimension $d$ and sample size $n$. Empirical evidence supports our finding that minimum-norm interpolants in RKHS can exhibit this unusual non-monotonicity in sample size; furthermore, locations of the peaks in our experiments match our theoretical predictions. Since gradient flow on appropriately initialized wide neural networks converges to a minimum-norm interpolant with respect to a certain kernel, our analysis also yields novel estimation and generalization guarantees for these over-parametrized models. At the heart of our analysis is a study of spectral properties of the random kernel matrix restricted to a filtration of eigen-spaces of the population covariance operator, and may be of independent interest.

Motivation & Objective

• To understand the generalization risk of minimum-norm interpolants in Reproducing Kernel Hilbert Spaces (RKHS).
• To characterize how the risk changes with respect to sample size n and input dimension d = n^α for α ∈ (0,1).
• To explain the emergence of non-monotonic, multiple-descent behavior in the risk curve.

Proposed method

• Analyzes the spectral properties of random kernel matrices restricted to a filtration of eigen-spaces of the population covariance operator.
• Derives upper bounds on the generalization risk using the structure of the kernel matrix and its eigen-decomposition.
• Studies the interplay between the kernel's spectral decay and the dimensionality scaling d = n^α.
• Uses a filtration of eigen-spaces to decompose the risk and isolate contributions from different frequency components.
• Applies the analysis to gradient flow on wide, appropriately initialized neural networks, linking them to minimum-norm interpolants via kernel equivalence.
• Employs theoretical bounds and empirical validation to confirm the predicted peaks in risk.

Experimental results

Research questions

• RQ1 How does the generalization risk of minimum-norm interpolants in RKHS behave as a function of sample size and input dimension scaling d = n^α for α ∈ (0,1)?
• RQ2 Why do minimum-norm interpolants exhibit non-monotonic, multiple-descent risk curves in high-dimensional settings?
• RQ3 What spectral properties of the kernel matrix govern the risk behavior in the over-parameterized regime?
• RQ4 How do the theoretical risk bounds compare to empirical observations in simulated or real data?

Key findings

• The risk of minimum-norm interpolants in RKHS exhibits a multiple-descent shape across different scalings of d = n^α for α ∈ (0,1).
• Empirical results confirm the presence of non-monotonic risk curves with peaks that align with theoretical predictions.
• The spectral structure of the kernel matrix, particularly its restriction to eigen-spaces of the population covariance, governs the risk behavior.
• The analysis provides novel generalization guarantees for over-parameterized neural networks trained via gradient flow, which converge to minimum-norm interpolants in an associated RKHS.
• The derived upper bounds on risk are non-monotonic and depend critically on the interplay between kernel eigen-decay and dimensionality scaling.
• The findings extend to wide neural networks through the kernel equivalence of gradient flow, offering new insights into their generalization properties.