[Paper Review] Quasi-Monte Carlo Feature Maps for Shift-Invariant Kernels

Haim Avron, Vikas Sindhwani+2 more|arXiv (Cornell University)|Dec 29, 2014

Mathematical Approximation and IntegrationMathematics35 references65 citations

TL;DR

This paper proposes using Quasi-Monte Carlo (QMC) sequences instead of Monte Carlo sampling to construct feature maps for shift-invariant kernels, significantly improving approximation accuracy and convergence speed. By minimizing a novel box discrepancy measure, the method achieves lower integration error and faster convergence in kernel approximation, outperforming standard randomized feature maps in both theory and practice.

ABSTRACT

We consider the problem of improving the efficiency of randomized Fourier feature maps to accelerate training and testing speed of kernel methods on large datasets. These approximate feature maps arise as Monte Carlo approximations to integral representations of shift-invariant kernel functions (e.g., Gaussian kernel). In this paper, we propose to use Quasi-Monte Carlo (QMC) approximations instead, where the relevant integrands are evaluated on a low-discrepancy sequence of points as opposed to random point sets as in the Monte Carlo approach. We derive a new discrepancy measure called box discrepancy based on theoretical characterizations of the integration error with respect to a given sequence. We then propose to learn QMC sequences adapted to our setting based on explicit box discrepancy minimization. Our theoretical analyses are complemented with empirical results that demonstrate the effectiveness of classical and adaptive QMC techniques for this problem.

Motivation & Objective

• To improve the efficiency and accuracy of randomized feature maps for shift-invariant kernels used in large-scale kernel methods.
• To replace traditional Monte Carlo sampling with Quasi-Monte Carlo (QMC) sequences to reduce integration error in kernel approximation.
• To introduce a new discrepancy measure—box discrepancy—tailored to the integration error in QMC-based feature map construction.
• To develop adaptive QMC sequences by minimizing box discrepancy for better performance on specific kernel functions.
• To demonstrate theoretically and empirically that QMC-based feature maps converge faster and with lower error than Monte Carlo counterparts.

Proposed method

• Proposes Quasi-Monte Carlo (QMC) approximation of integral representations of shift-invariant kernels, replacing Monte Carlo sampling with low-discrepancy sequences.
• Introduces a new discrepancy measure called box discrepancy, derived from theoretical characterizations of integration error in kernel approximation.
• Derives a closed-form expression for the expected squared box discrepancy of a QMC sequence, enabling optimization-based sequence learning.
• Proposes to learn adaptive QMC sequences by minimizing the box discrepancy, improving convergence for specific kernel functions.
• Uses the method to construct feature maps for shift-invariant kernels such as the Gaussian kernel, with theoretical guarantees on approximation error.
• Employs complex-valued feature maps and leverages the Bochner representation to connect kernel functions to Fourier transforms.

Experimental results

Research questions

• RQ1 Can Quasi-Monte Carlo (QMC) sequences reduce integration error in randomized feature map construction for shift-invariant kernels compared to Monte Carlo sampling?
• RQ2 What is a suitable discrepancy measure for evaluating QMC sequences in the context of kernel approximation, and how can it be minimized?
• RQ3 How does the proposed box discrepancy measure compare to existing discrepancy measures in terms of bounding integration error for kernel feature maps?
• RQ4 Can adaptive QMC sequences, learned via box discrepancy minimization, improve the convergence rate and accuracy of kernel approximation?
• RQ5 What empirical improvements in training and testing speed and accuracy can be achieved by replacing Monte Carlo with QMC in randomized feature maps?

Key findings

• The proposed QMC-based feature maps achieve lower integration error than Monte Carlo methods due to the use of low-discrepancy sequences instead of random sampling.
• The introduction of box discrepancy provides a theoretically grounded measure for evaluating QMC sequences in kernel approximation, with a closed-form expression for expected squared discrepancy.
• Adaptive QMC sequences learned via box discrepancy minimization outperform classical QMC sequences and Monte Carlo sampling in terms of convergence speed and approximation accuracy.
• Empirical results demonstrate that QMC-based feature maps achieve comparable or better test accuracy with fewer random features than Monte Carlo counterparts.
• The method reduces the number of required random features for a given approximation error, leading to faster training and prediction times in large-scale kernel methods.
• Theoretical analysis shows that the expected squared error of the QMC approximation decays at a rate of $O(1/s)$, matching the Monte Carlo rate but with a smaller constant due to lower discrepancy.