[Paper Review] To understand deep learning we need to understand kernel learning

Mikhail Belkin, Siyuan Ma+1 more|arXiv (Cornell University)|Feb 5, 2018

Face and Expression RecognitionComputer Science28 references110 citations

TL;DR

The paper shows that overfitted and interpolated kernel methods generalize well on real and synthetic data, drawing parallels with deep networks, while existing generalization bounds fail to explain this behavior.

ABSTRACT

Generalization performance of classifiers in deep learning has recently become a subject of intense study. Deep models, typically over-parametrized, tend to fit the training data exactly. Despite this "overfitting", they perform well on test data, a phenomenon not yet fully understood. The first point of our paper is that strong performance of overfitted classifiers is not a unique feature of deep learning. Using six real-world and two synthetic datasets, we establish experimentally that kernel machines trained to have zero classification or near zero regression error perform very well on test data, even when the labels are corrupted with a high level of noise. We proceed to give a lower bound on the norm of zero loss solutions for smooth kernels, showing that they increase nearly exponentially with data size. We point out that this is difficult to reconcile with the existing generalization bounds. Moreover, none of the bounds produce non-trivial results for interpolating solutions. Second, we show experimentally that (non-smooth) Laplacian kernels easily fit random labels, a finding that parallels results for ReLU neural networks. In contrast, fitting noisy data requires many more epochs for smooth Gaussian kernels. Similar performance of overfitted Laplacian and Gaussian classifiers on test, suggests that generalization is tied to the properties of the kernel function rather than the optimization process. Certain key phenomena of deep learning are manifested similarly in kernel methods in the modern "overfitted" regime. The combination of the experimental and theoretical results presented in this paper indicates a need for new theoretical ideas for understanding properties of classical kernel methods. We argue that progress on understanding deep learning will be difficult until more tractable "shallow" kernel methods are better understood.

Motivation & Objective

• Demonstrate that overfitted/interpolated kernel classifiers can generalize well on diverse datasets.
• Show that non-smooth kernels (Laplacian) can fit random labels while Gaussian kernels are harder to fit yet yield similar test performance.
• Provide theoretical lower bounds showing RKHS norms of interpolating solutions grow rapidly with data size under nonzero label noise.
• Argue that current kernel/generalization bounds do not capture interpolated kernel behavior and that new theory is needed.
• Highlight the relationship between kernel structure and generalization, independent of optimization dynamics.

Proposed method

• Use kernel machines in RKHS as linear regression in infinite dimensions with Gaussian and Laplacian kernels.
• Construct interpolating solutions via Representer Theorem and solve for alpha in K alpha = y (Eq. 2).
• Compare overfitted (zero classification error) and interpolated (zero regression loss) solutions across multiple datasets.
• Employ EigenPro-SGD as an accelerated kernel learning method to reach zero classification error.
• Theoretically derive a lower bound showing the RKHS norm must grow nearly exponentially with data size for t-overfitting with nonzero label noise.

Experimental results

Research questions

• RQ1 Do overfitted/interpolated kernel methods generalize well on real-world and synthetic datasets?
• RQ2 How do smooth (Gaussian) versus non-smooth (Laplacian) kernels compare in fitting noisy or random labels and in test performance?
• RQ3 Why do existing generalization bounds fail to explain the performance of interpolated kernel classifiers, and what theory could better describe it?

Key findings

• Interpolating kernel classifiers achieve near-optimal test performance even with high label noise across six real-world and two synthetic datasets.
• Regularization by early stopping provides at most a small improvement in test performance for these interpolated classifiers.
• Non-smooth Laplacian kernels can fit random labels easily, mirroring observations in ReLU networks; smooth Gaussian kernels require more epochs to fit noisy data.
• The RKHS norm of overfitted solutions for smooth kernels grows nearly exponentially with data size, diverging from typical bounds that depend polynomially on the norm.
• Empirical test performance of interpolated kernel classifiers remains robust and is similar between Laplacian and Gaussian kernels despite added label noise.