[Paper Review] For interpolating kernel machines, minimizing the norm of the ERM solution minimizes stability

Akshay Rangamani, Lorenzo Rosasco+1 more|arXiv (Cornell University)|Jun 28, 2020

Sparse and Compressive Sensing Techniques25 references4 citations

TL;DR

This paper establishes that for interpolating kernel machines, minimizing the norm of the empirical risk minimization (ERM) solution minimizes a bound on cross-validation stability, which in turn controls generalization error. The key result shows that the minimum norm interpolating solution achieves optimal stability and risk bounds, linking numerical stability (condition number of the kernel matrix) to statistical stability in overparameterized settings.

ABSTRACT

We study the average $\mbox{CV}_{loo}$ stability of kernel ridge-less regression and derive corresponding risk bounds. We show that the interpolating solution with minimum norm minimizes a bound on $\mbox{CV}_{loo}$ stability, which in turn is controlled by the condition number of the empirical kernel matrix. The latter can be characterized in the asymptotic regime where both the dimension and cardinality of the data go to infinity. Under the assumption of random kernel matrices, the corresponding test error should be expected to follow a double descent curve.

Motivation & Objective

• To understand generalization in overparameterized kernel methods where models perfectly fit training data (interpolation) without regularization.
• To characterize the stability of interpolating solutions in kernel least squares using a stability-based approach.
• To show that among all interpolating solutions, the minimum norm solution minimizes a bound on cross-validation stability.
• To establish a connection between the condition number of the empirical kernel matrix and both numerical and statistical stability.

Proposed method

• Analyzes the stability of interpolating solutions in kernel least squares using leave-one-out cross-validation (CV_loo) stability as a measure.
• Derives an upper bound on CV_loo stability that depends on the condition number of the empirical kernel matrix.
• Uses matrix inequalities and random matrix theory to analyze the asymptotic behavior of the kernel matrix as both sample size n and dimension d grow large.
• Shows that the minimum norm solution minimizes the derived stability bound, implying better generalization.
• Connects the stability bound to the pseudoinverse of the kernel matrix, linking numerical and statistical stability.
• Demonstrates that gradient descent converges to the minimum norm solution in linear kernel settings, making the results applicable to optimization dynamics.

Experimental results

Research questions

• RQ1 Does minimizing the norm of the ERM solution lead to better stability and generalization in interpolating kernel machines?
• RQ2 How is the stability of interpolating solutions related to the condition number of the empirical kernel matrix?
• RQ3 Can stability-based bounds explain the double descent behavior in test error observed in overparameterized kernel models?
• RQ4 Is there a link between numerical stability (condition number) and statistical stability (CV_loo) in kernel methods?
• RQ5 How does the minimum norm solution compare to other interpolating solutions in terms of risk and stability?

Key findings

• The minimum norm interpolating solution minimizes an upper bound on CV_loo stability, implying it has the best generalization performance among all interpolating solutions.
• The stability bound is controlled by the condition number of the empirical kernel matrix, which governs both numerical and statistical stability.
• In the asymptotic regime where n and d grow large, the condition number of the kernel matrix follows a double descent curve, consistent with empirical observations.
• The risk bounds for the minimum norm solution depend on the pseudoinverse of the kernel matrix, establishing a direct link between numerical and statistical stability.
• Gradient descent converges to the minimum norm solution in linear kernel settings, meaning the stability benefits apply to standard training procedures.
• The results provide a theoretical foundation for the success of ridgeless kernel methods in overparameterized regimes, explaining generalization without explicit regularization.