[Paper Review] Data-dependent compression of random features for large-scale kernel approximation

Raj Agrawal, Trevor Campbell+2 more|arXiv (Cornell University)|Oct 9, 2018

Stochastic Gradient Optimization Techniques13 citations

TL;DR

This paper proposes a data-dependent compression method that reduces a large set of random features to a small, weighted subset while preserving the theoretical guarantees of kernel approximation. By combining random feature maps with efficient, data-driven feature selection, the method achieves near-optimal kernel matrix approximation using only O(log J₊) features—significantly fewer than state-of-the-art methods—demonstrated on datasets with over 50 million observations.

ABSTRACT

Kernel methods offer the flexibility to learn complex relationships in modern, large data sets while enjoying strong theoretical guarantees on quality. Unfortunately, these methods typically require cubic running time in the data set size, a prohibitive cost in the large-data setting. Random feature maps (RFMs) and the Nystrom method both consider low-rank approximations to the kernel matrix as a potential solution. But, in order to achieve desirable theoretical guarantees, the former may require a prohibitively large number of features J+, and the latter may be prohibitively expensive for high-dimensional problems. We propose to combine the simplicity and generality of RFMs with a data-dependent feature selection scheme to achieve desirable theoretical approximation properties of Nystrom with just O(log J+) features. Our key insight is to begin with a large set of random features, then reduce them to a small number of weighted features in a data-dependent, computationally efficient way, while preserving the statistical guarantees of using the original large set of features. We demonstrate the efficacy of our method with theory and experiments--including on a data set with over 50 million observations. In particular, we show that our method achieves small kernel matrix approximation error and better test set accuracy with provably fewer random features than state-of-the-art methods.

Motivation & Objective

• To address the high computational cost of kernel methods in large-scale settings by reducing the number of random features needed for accurate kernel approximation.
• To combine the theoretical robustness of Nyström methods with the simplicity and generality of random feature maps.
• To develop a computationally efficient, data-dependent feature selection scheme that maintains statistical guarantees of the original large feature set.
• To achieve provably better approximation error and test accuracy than existing methods with substantially fewer features.

Proposed method

• Start with a large set of J₊ random features to ensure theoretical approximation guarantees.
• Apply a data-dependent compression scheme to select a small, weighted subset of features from the initial set.
• Use a convex optimization framework to identify the most informative features based on data geometry and kernel structure.
• Leverage the convex hull and distance-based criteria to prioritize features that best represent the data manifold.
• Ensure the compressed feature set maintains the same statistical properties as the full set, including kernel matrix approximation error bounds.
• Optimize the selection process to scale efficiently to large datasets, such as those with over 50 million observations.

Experimental results

Research questions

• RQ1 Can a data-dependent compression scheme reduce the number of random features needed for kernel approximation while preserving theoretical guarantees?
• RQ2 How does the proposed method compare to Nyström and standard random feature maps in terms of approximation error and test accuracy?
• RQ3 What is the minimal number of features required to achieve near-optimal kernel approximation under data-dependent selection?
• RQ4 Can the method scale to massive datasets, such as those with over 50 million observations, while maintaining low computational cost?

Key findings

• The method achieves kernel matrix approximation error comparable to full random feature maps but with only O(log J₊) features, significantly reducing the feature count.
• On a dataset with over 50 million observations, the method maintains high test set accuracy while using far fewer features than state-of-the-art approaches.
• The data-dependent compression scheme preserves the theoretical approximation guarantees of the original large feature set, ensuring robust performance.
• The method outperforms standard random feature maps and Nyström-based methods in both approximation error and test accuracy with substantially reduced feature dimensionality.