[Paper Review] Learning Infinite-Layer Networks: Without the Kernel Trick

Roi Livni, Daniel Carmon+1 more|arXiv (Cornell University)|Jun 16, 2016

Advanced Bandit Algorithms Research33 references3 citations

TL;DR

This paper presents an online learning algorithm for Infinite Layer Networks (ILNs) that bypasses the kernel trick by using random feature sampling, achieving regret bounds matching those of kernel methods. The key contribution is proving that random features suffice for efficient learning even when the kernel cannot be computed analytically, with sample complexity comparable to kernel-based approaches.

ABSTRACT

Infinite--Layer Networks (ILN) have recently been proposed as an architecture that mimics neural networks while enjoying some of the advantages of kernel methods. ILN are networks that integrate over infinitely many nodes within a single hidden layer. It has been demonstrated by several authors that the problem of learning ILN can be reduced to the kernel trick, implying that whenever a certain integral can be computed analytically they are efficiently learnable. In this work we give an online algorithm for ILN, which avoids the kernel trick assumption. More generally and of independent interest, we show that kernel methods in general can be exploited even when the kernel cannot be efficiently computed but can only be estimated via sampling. We provide a regret analysis for our algorithm, showing that it matches the sample complexity of methods which have access to kernel values. Thus, our method is the first to demonstrate that the kernel trick is not necessary as such, and random features suffice to obtain comparable performance.

Motivation & Objective

• To develop an efficient learning algorithm for Infinite Layer Networks (ILNs) that does not rely on the kernel trick.
• To show that random feature sampling can replace exact kernel computation in ILN learning while maintaining comparable performance.
• To establish theoretical regret bounds for the proposed method that match those of kernel-based approaches.
• To demonstrate that random features are sufficient for achieving optimal sample complexity in ILN learning.

Proposed method

• The algorithm uses online stochastic gradient descent with unbiased estimates of the kernel function via random sampling of parameters w from the prior measure μ.
• It constructs a random feature approximation of the integral in the ILN formulation by sampling w and computing ψ(x;w) for each sample.
• The method employs a normalization scheme to ensure bounded feature norms and stability in gradient updates.
• A regret analysis is conducted to show that the algorithm achieves O(1/√T) regret, matching the theoretical lower bound for supervised learning.
• The algorithm dynamically adjusts the weight vector f_t based on prediction error, with pruning steps to maintain computational efficiency.
• Theoretical bounds are derived using concentration inequalities to control estimation error from finite sampling of features.

Experimental results

Research questions

• RQ1 Can Infinite Layer Networks be learned efficiently without relying on the kernel trick?
• RQ2 Is random feature sampling sufficient to achieve optimal regret and sample complexity in ILN learning?
• RQ3 What is the relationship between the number of sampled features and the generalization performance of ILNs?
• RQ4 Can online learning algorithms for ILNs match the theoretical performance of batch kernel methods?

Key findings

• The proposed algorithm achieves a regret bound of O(1/√T), matching the theoretical lower bound for supervised learning.
• The method requires Õ(T²) calls to the random feature generator, which is comparable to the cost of kernel matrix computation in batch methods.
• The algorithm maintains performance comparable to kernel-based methods even when the kernel cannot be computed analytically.
• Theoretical analysis shows that random features provide an unbiased estimate of the kernel, enabling convergence with optimal sample complexity.
• The algorithm's performance is robust under bounded feature norms and controlled gradient variance, as shown by concentration bounds.
• The method improves upon prior random feature approaches in terms of sample complexity and expressiveness for ILN learning.