[Paper Review] Random Feature Maps for Dot Product Kernels

Purushottam Kar, Harish Karnick|arXiv (Cornell University)|Jan 31, 2012

Face and Expression Recognition14 references121 citations

TL;DR

This paper proposes a novel randomized feature map for dot product kernels, leveraging harmonic analysis to embed non-linear feature spaces into low-dimensional Euclidean spaces with low distortion. By extending Rahimi and Recht's work beyond translation-invariant kernels, it achieves high-accuracy classification with significantly reduced training and testing times using fewer random features compared to baseline methods.

ABSTRACT

Approximating non-linear kernels using feature maps has gained a lot of interest in recent years due to applications in reducing training and testing times of SVM classifiers and other kernel based learning algorithms. We extend this line of work and present low distortion embeddings for dot product kernels into linear Euclidean spaces. We base our results on a classical result in harmonic analysis characterizing all dot product kernels and use it to define randomized feature maps into explicit low dimensional Euclidean spaces in which the native dot product provides an approximation to the dot product kernel with high confidence.

Motivation & Objective

• To address the 'curse of support' in kernel methods, where large support sets slow down prediction due to expensive kernel evaluations.
• To extend randomized feature mapping beyond translation-invariant kernels to general dot product kernels using harmonic analysis.
• To develop an explicit, low-dimensional embedding that preserves kernel similarity with high probability.
• To improve efficiency of kernel methods by replacing implicit kernel computations with explicit feature maps in low-dimensional space.

Proposed method

• Uses a classical result in harmonic analysis to characterize all dot product kernels as inner products of feature maps.
• Constructs a randomized feature map Z: Rd → RD such that ⟨Z(x), Z(y)⟩ ≈ K(x, y) with high probability.
• Employs a modified feature map H0/1 that adds a constant bias term to improve approximation accuracy.
• Derives the distribution of random features based on the Fourier transform of the kernel function.
• Applies the Johnson-Lindenstrauss lemma to ensure low distortion in inner product approximation.
• Employs a two-stage construction: first sampling from the Fourier transform of the kernel, then applying a bias correction via H0/1.

Experimental results

Research questions

• RQ1 Can we extend random feature maps beyond translation-invariant kernels to general dot product kernels?
• RQ2 How can we achieve low-distortion embeddings of dot product kernels into low-dimensional Euclidean spaces?
• RQ3 What is the impact of adding a bias term (H0/1) on approximation accuracy and computational efficiency?
• RQ4 How does the number of random features affect classification accuracy and training/test speed in practice?

Key findings

• The H0/1 feature map achieves significantly higher classification accuracy than the standard random feature map (RF) when using a small number of random features.
• On average, H0/1 requires only d = 45 additional features (where d is input dimension), yet delivers superior performance.
• For all four benchmark datasets (Spambase, Nursery, IJCNN, Cod-RNA), H0/1 outperforms RF in accuracy at low D, with the gap narrowing only as D increases.
• Training and testing times are dramatically reduced with H0/1, especially at low D, due to faster feature construction and improved separability.
• The method maintains competitive classification accuracy while reducing training time by up to several orders of magnitude in some cases.
• The feature map construction is stable and generalizable across different kernel types, including polynomial and exponential kernels.