[Paper Review] A la Carte - Learning Fast Kernels

Zichao Yang, Alexander J. Smola+2 more|arXiv (Cornell University)|Dec 19, 2014

Gaussian Processes and Bayesian Inference14 references17 citations

TL;DR

This paper introduces a family of fast, flexible, and lightly parametrized kernel learning methods based on Fastfood basis function expansions, where spectral frequencies are learned in groups to enable scalable and expressive kernel approximation. The method achieves state-of-the-art accuracy with significantly reduced memory and training time compared to existing approaches, even outperforming models with orders of magnitude more parameters.

ABSTRACT

Kernel methods have great promise for learning rich statistical representations of large modern datasets. However, compared to neural networks, kernel methods have been perceived as lacking in scalability and flexibility. We introduce a family of fast, flexible, lightly parametrized and general purpose kernel learning methods, derived from Fastfood basis function expansions. We provide mechanisms to learn the properties of groups of spectral frequencies in these expansions, which require only O(mlogd) time and O(m) memory, for m basis functions and d input dimensions. We show that the proposed methods can learn a wide class of kernels, outperforming the alternatives in accuracy, speed, and memory consumption.

Motivation & Objective

• To address the scalability and flexibility trade-off in kernel methods for large modern datasets.
• To enable automatic learning of kernel properties without requiring prior knowledge of the optimal kernel form.
• To develop a general-purpose kernel learning framework that is both expressive and computationally efficient.
• To reduce overfitting and computational burden in high-dimensional kernel learning by grouping spectral frequencies.
• To outperform existing kernel methods in accuracy, speed, and memory consumption on diverse real-world datasets.

Proposed method

• Leverages Fastfood basis function expansions to approximate kernel functions with O(m log d) time and O(m) memory, where m is the number of basis functions and d is input dimensionality.
• Introduces a mechanism to learn the scales, spread, and locations of groups of spectral frequencies via marginal likelihood optimization.
• Proposes four new kernel learning methods: Gaussian Mixture (GM), Piecewise Linear (PWL), Fastfood Spectral Basis with ARD (FSGBARD), and Fastfood Spectral Basis with RBF (FSARD).
• Uses grouped frequency learning to allow arbitrarily many basis functions without overfitting, maintaining low parametrization.
• Applies weighted mixtures of Fastfood expansions to learn flexible, translation-invariant kernels while preserving computational efficiency.
• Employs marginal likelihood optimization to tune group-level frequency parameters, enabling adaptive kernel learning with minimal free parameters.

Experimental results

Research questions

• RQ1 Can spectral frequencies in Fastfood expansions be learned adaptively to improve kernel flexibility without increasing computational cost?
• RQ2 Can grouped learning of frequency properties reduce overfitting and enable scalable kernel learning with high expressiveness?
• RQ3 How does the performance of the proposed methods compare to exact kernel methods and existing fast approximations in terms of accuracy, speed, and memory?
• RQ4 Can the proposed method outperform models with significantly more parameters, such as SSGPR, despite having a much lighter parametrization?
• RQ5 Does the ability to learn group-level frequency properties lead to better generalization and scalability on large, diverse datasets?

Key findings

• The GM and PWL models outperform all alternatives in predictive accuracy, even when SSGPR has orders of magnitude more free parameters.
• Despite having far fewer parameters than SSGPR, GM and PWL achieve comparable or better accuracy with significantly lower memory and training time.
• FSGBARD matches SSGPR's accuracy but requires substantially less memory and runtime due to its Fastfood-based representation.
• The performance of GM and PWL continues to improve with more basis functions, unlike SSGPR and FSGBARD, which suffer from overfitting as parameters increase.
• On large datasets (n > 2000), exact RBF and ARD kernels become intractable, but the proposed methods remain scalable and highly accurate.
• The proposed methods achieve state-of-the-art performance in accuracy, training time, testing time, and memory consumption across a wide range of real-world datasets.