[Paper Review] Data-driven Random Fourier Features using Stein Effect

Wei-Cheng Chang, Chunliang Li+2 more|arXiv (Cornell University)|May 23, 2017

Gaussian Processes and Bayesian Inference23 references3 citations

TL;DR

This paper proposes a data-driven random Fourier feature weighting method using the Stein effect to reduce kernel approximation error. By learning non-uniform weights via a shrinkage estimator that minimizes empirical risk, the method outperforms uniform-weighted MC and QMC baselines in kernel approximation and supervised learning tasks across six benchmark datasets.

ABSTRACT

Large-scale kernel approximation is an important problem in machine learning research. Approaches using random Fourier features have become increasingly popular [Rahimi and Recht, 2007], where kernel approximation is treated as empirical mean estimation via Monte Carlo (MC) or Quasi-Monte Carlo (QMC) integration [Yang et al., 2014]. A limitation of the current approaches is that all the features receive an equal weight summing to 1. In this paper, we propose a novel shrinkage estimator from "Stein effect", which provides a data-driven weighting strategy for random features and enjoys theoretical justifications in terms of lowering the empirical risk. We further present an efficient randomized algorithm for large-scale applications of the proposed method. Our empirical results on six benchmark data sets demonstrate the advantageous performance of this approach over representative baselines in both kernel approximation and supervised learning tasks.

Motivation & Objective

• Address the sub-optimality of uniform weighting in random Fourier features for kernel approximation.
• Develop a data-driven weighting strategy that reduces empirical risk in kernel approximation without requiring label information.
• Provide theoretical justification via the Stein effect for improved estimation over empirical mean estimators.
• Design an efficient randomized algorithm suitable for large-scale applications.
• Demonstrate consistent performance gains over MC, QMC, and Bayesian Quadrature baselines in both kernel approximation and supervised learning tasks.

Proposed method

• Leverage the Stein effect to derive a shrinkage estimator that learns non-uniform weights for random Fourier features.
• Formulate the kernel approximation as a convex optimization problem to minimize empirical risk with data-driven weights.
• Introduce a randomized sketching-based solver that samples only O(M) data points to approximate the optimal weight vector, ensuring scalability.
• Extend Bayesian Quadrature (BQ) to kernel approximation as a baseline, but avoid its sensitivity to hyperparameter tuning by using a data-driven approach.
• Use Monte Carlo (MC) and Quasi-Monte Carlo (QMC) methods as uniform-weighted baselines for comparison.
• Apply the proposed method to shift-invariant kernels via Bochner's theorem, enabling efficient feature generation and weight learning.

Experimental results

Research questions

• RQ1 Can a data-driven weighting strategy for random Fourier features reduce kernel approximation error compared to uniform weighting?
• RQ2 Does the Stein effect provide a theoretically justified framework for learning optimal feature weights in kernel approximation?
• RQ3 How does the proposed method perform in comparison to MC, QMC, and BQ baselines in terms of kernel approximation and downstream supervised learning?
• RQ4 What is the computational efficiency of the randomized solver in large-scale settings, and how many samples are needed for convergence?
• RQ5 How do non-uniform feature weights compare to uniform weights in reducing empirical risk across different datasets and feature counts?

Key findings

• The proposed Stein-Effect Shrinkage (SES) estimator consistently reduces kernel approximation error across six benchmark datasets compared to uniform-weighted MC and QMC methods.
• SES outperforms or matches the best baseline in most supervised learning tasks (classification and regression), demonstrating transferability of improved kernel approximation.
• The randomized solver requires only 2–4 times the number of random features (M) in sampled data to achieve stable and accurate weight estimation, confirming theoretical scalability.
• As the number of random features increases (M = 32 to 512), the distribution of learned weights in SES and BQ converges toward uniformity, aligning with bias-variance trade-off theory.
• Non-uniform shrinkage weights in SES yield significantly lower empirical risk than uniform weights, confirming the benefit of data-driven adaptation.
• Bayesian Quadrature (BQ) shows inconsistent performance with respect to sampled data size and is highly sensitive to hyperparameter tuning, unlike the proposed method.