[Paper Review] Scalable Kernel Methods via Doubly Stochastic Gradients

Bo Dai, Bo Xie+5 more|arXiv (Cornell University)|Jul 21, 2014

Stochastic Gradient Optimization TechniquesComputer Science38 references74 citations

TL;DR

This paper proposes Doubly Stochastic Functional Gradients (DSFG) to scale kernel methods for large datasets by combining random training samples and random features to approximate functional gradients in kernel ridge regression and classification. The method achieves O(1/t) convergence rate and O(1/√t) generalization error, matching neural networks in performance on 2.3M molecular, 8M MNIST, and 1M ImageNet datasets while reducing memory from O(n²) to O(n).

ABSTRACT

The general perception is that kernel methods are not scalable, and neural nets are the methods of choice for nonlinear learning problems. Or have we simply not tried hard enough for kernel methods? Here we propose an approach that scales up kernel methods using a novel concept called "doubly stochastic functional gradients". Our approach relies on the fact that many kernel methods can be expressed as convex optimization problems, and we solve the problems by making two unbiased stochastic approximations to the functional gradient, one using random training points and another using random functions associated with the kernel, and then descending using this noisy functional gradient. We show that a function produced by this procedure after $t$ iterations converges to the optimal function in the reproducing kernel Hilbert space in rate $O(1/t)$, and achieves a generalization performance of $O(1/\sqrt{t})$. This doubly stochasticity also allows us to avoid keeping the support vectors and to implement the algorithm in a small memory footprint, which is linear in number of iterations and independent of data dimension. Our approach can readily scale kernel methods up to the regimes which are dominated by neural nets. We show that our method can achieve competitive performance to neural nets in datasets such as 8 million handwritten digits from MNIST, 2.3 million energy materials from MolecularSpace, and 1 million photos from ImageNet.

Motivation & Objective

• Address the scalability bottleneck of kernel methods due to O(n²) memory and computation from dense kernel matrices.
• Overcome limitations of low-rank and random feature approximations that degrade generalization unless rank/feature count scales with n.
• Enable streaming learning with growing function class flexibility as new data arrives.
• Develop a simple, efficient optimization framework for kernel methods that avoids storing all support vectors.
• Achieve optimal convergence and generalization rates without sacrificing statistical performance.

Proposed method

• Express kernel methods as convex optimization over functions in a reproducing kernel Hilbert space (RKHS).
• Approximate the functional gradient using two independent sources of randomness: random training points and random features associated with the kernel.
• Use unbiased stochastic approximations to the functional gradient, enabling efficient functional gradient descent with O(n) memory and O(nrd) computation per iteration.
• Dynamically grow the number of random features as data arrives, allowing flexible model capacity in streaming settings.
• Leverage the double stochasticity to maintain convergence guarantees while avoiding explicit kernel matrix storage.
• Support extensions via Fastfood, Quasi-Monte Carlo, or adaptive feature learning via back-propagation.

Experimental results

Research questions

• RQ1 Can kernel methods achieve comparable performance to deep neural networks on large-scale nonlinear learning tasks?
• RQ2 Can we design a scalable kernel method that avoids the O(n²) memory and computation of traditional kernel methods?
• RQ3 Does a doubly stochastic functional gradient approach maintain optimal convergence and generalization rates despite two sources of noise?
• RQ4 Can the method adaptively grow model capacity in streaming settings without retraining?
• RQ5 How does the performance of the proposed method compare to neural networks on real-world large-scale datasets?

Key findings

• The proposed DSFG method achieves an O(1/t) convergence rate to the optimal function in the RKHS, matching the best-known rate for stochastic convex optimization.
• Generalization error converges at O(1/√t), independent of input dimension, demonstrating strong statistical performance.
• On the 2.3 million molecular dataset from MolecularSpace, the method achieved a Mean Absolute Error (MAE) of 2.97 kcal/mol, outperforming neural networks (3.51 kcal/mol).
• On the 8 million MNIST dataset, the method matched neural network performance with faster convergence.
• On the 1 million ImageNet dataset using convolutional features, the method achieved competitive accuracy and speed, surpassing neural nets with pre-learned features.
• The method reduces memory usage from O(n²) to O(n), enabling practical deployment on large-scale datasets previously dominated by neural networks.