[Paper Review] A representer theorem for deep kernel learning

Bastian Bohn, Michael Griebel+1 more|arXiv (Cornell University)|Sep 29, 2017

Model Reduction and Neural Networks3 citations

TL;DR

This paper establishes a representer theorem for deep kernel learning, proving that optimal solutions to nonlinear, infinite-dimensional minimization problems in multi-layer kernel networks can be expressed as finite-dimensional nonlinear functions of the data. The key contribution is a theoretical foundation enabling the use of standard nonlinear optimization algorithms on deep kernel models, with direct applications to deep learning and multi-layer multiple kernel learning methods.

ABSTRACT

In this paper we provide a finite-sample and an infinite-sample representer theorem for the concatenation of (linear combinations of) kernel functions of reproducing kernel Hilbert spaces. These results serve as mathematical foundation for the analysis of machine learning algorithms based on compositions of functions. As a direct consequence in the finite-sample case, the corresponding infinite-dimensional minimization problems can be recast into (nonlinear) finite-dimensional minimization problems, which can be tackled with nonlinear optimization algorithms. Moreover, we show how concatenated machine learning problems can be reformulated as neural networks and how our representer theorem applies to a broad class of state-of-the-art deep learning methods.

Motivation & Objective

• To address the lack of a theoretical framework for analyzing deep kernel learning and multi-layer multiple kernel learning (MLMKL) methods.
• To extend the classical representer theorem—originally for single-layer kernel methods—to deep, concatenated kernel architectures.
• To provide a mathematical foundation that reduces infinite-dimensional optimization problems in deep kernel networks to finite-dimensional ones.
• To demonstrate the applicability of the representer theorem to state-of-the-art deep learning models, including deep SVMs and neural networks with learned kernels.
• To enable the use of standard nonlinear optimization techniques for training deep kernel models by proving the existence of a finite-dimensional solution representation.

Proposed method

• Formalizes the optimal concatenated approximation problem in reproducing kernel Hilbert spaces (RKHS) using arbitrary loss functions and regularizers.
• Derives a representer theorem for multi-layer kernel networks by proving that critical points of the objective functional admit a finite-dimensional representation in terms of kernel evaluations on training data.
• Applies the theorem to both finite-sample and infinite-sample settings, showing that the solution lies in a finite-dimensional subspace spanned by kernel functions evaluated at input points.
• Uses integral formulations and the reproducing property of RKHS kernels to express solutions as weighted sums of kernel functions, with weights determined by the loss and data distribution.
• Establishes conditions under which the loss function and its derivatives are integrable and differentiable, ensuring the validity of the representer theorem in deep architectures.
• Demonstrates that the derived solution structure naturally generalizes to deep neural networks and deep SVMs by interpreting the kernel composition as a hierarchical feature learning mechanism.

Experimental results

Research questions

• RQ1 Can a representer theorem be established for deep kernel learning models that are composed of multiple layers of kernel functions?
• RQ2 Under what conditions can the solution to an infinite-dimensional minimization problem in a deep kernel network be represented in a finite-dimensional subspace?
• RQ3 How does the proposed representer theorem extend classical results from single-layer kernel methods to multi-layer architectures?
• RQ4 In what way does the theorem enable the application of standard nonlinear optimization algorithms to deep kernel models?
• RQ5 What is the theoretical connection between deep kernel learning, multi-layer multiple kernel learning (MLMKL), and deep neural networks?

Key findings

• The paper proves a finite-sample and infinite-sample representer theorem for deep kernel learning, showing that optimal solutions to the minimization problem can be represented as finite-dimensional nonlinear functions of the training data.
• The solution structure allows the reformulation of the original infinite-dimensional optimization problem into a finite-dimensional one, making it amenable to standard nonlinear optimization techniques.
• The theorem applies to a broad class of deep learning models, including deep neural networks and deep SVMs, when interpreted as composed kernel functions.
• The solution for each layer is expressed as an integral involving the kernel and the gradient of the loss function, which can be discretized for numerical computation.
• The proof relies on the integrability and differentiability of the loss function and its derivatives, with bounds derived using the chain rule and dominated convergence theorem.
• The result generalizes the RLS2 method from [9] to non-linear outer kernels, providing a theoretical basis for multi-layer kernel learning with improved flexibility and approximation power.