[Paper Review] Every Model Learned by Gradient Descent Is Approximately a Kernel Machine

Pedro Domingos|arXiv (Cornell University)|Nov 30, 2020

Domain Adaptation and Few-Shot Learning37 references49 citations

TL;DR

The paper proves that any model trained with gradient descent is approximately a kernel machine using a path kernel, making deep networks interpretable as kernel methods.

ABSTRACT

Deep learning's successes are often attributed to its ability to automatically discover new representations of the data, rather than relying on handcrafted features like other learning methods. We show, however, that deep networks learned by the standard gradient descent algorithm are in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel). This greatly enhances the interpretability of deep network weights, by elucidating that they are effectively a superposition of the training examples. The network architecture incorporates knowledge of the target function into the kernel. This improved understanding should lead to better learning algorithms.

Motivation & Objective

• Motivate understanding of why deep networks trained by gradient descent behave like kernel machines.
• Show that the end model can be expressed as a kernel machine with a path kernel.
• Explain the interpretability implications of deep networks as superpositions of training examples in gradient space.

Proposed method

• Define tangent and path kernels as the basis for the gradient flow during learning.
• Derive that, as the learning rate tends to zero, the final model equals a kernel machine with a path kernel.
• Express the final model as y = sum_i a_i K(x, x_i) + b, where K is the path kernel.
• Relate the result to neural tangent kernels and show positive definiteness under path integration.
• Discuss how gradient descent acts like boosting over a sequence of tangent kernels.

Experimental results

Research questions

• RQ1 Can every gradient-descent-trained model be represented as a kernel machine with a data-dependent kernel?
• RQ2 How is the kernel defined in terms of the learning path and weight gradients?
• RQ3 What are the interpretability and scalability implications of viewing deep networks as kernel machines?
• RQ4 How does this view relate to neural tangent kernels and existing kernel theory?
• RQ5 What are the limitations and open questions for non-small learning rates or non-gradient-based methods?

Key findings

• Gradient descent yields models that are approximately kernel machines with a path kernel.
• The path kernel integrates the dot product of gradients along the learning path.
• The final model is a sum of a prior model and a kernel-based correction dependent on training data.
• For linear models, the path kernel reduces to the dot product, recovering the perceptron as a kernel machine.
• Regularization and loss derivatives weight training examples through the kernel, influencing their impact.
• The result provides interpretability by showing weights as a superposition of training examples in gradient space.