[Paper Review] Towards A Unified Analysis of Random Fourier Features

Zhu Li, Jean-François Ton+2 more|Research Portal (King's College London)|Jun 24, 2018

Sparse and Compressive Sensing Techniques39 references4 citations

TL;DR

This paper presents the first unified risk analysis for learning with random Fourier features using squared error and Lipschitz continuous loss functions. It establishes problem-specific trade-offs between computational cost and risk convergence rate in terms of regularization and effective degrees of freedom, improving existing bounds and introducing a data-dependent sampling method based on ridge leverage scores with a provably efficient approximation scheme.

ABSTRACT

Random Fourier features is a widely used, simple, and effective technique for scaling up kernel methods. The existing theoretical analysis of the approach, however, remains focused on specific learning tasks and typically gives pessimistic bounds which are at odds with the empirical results. We tackle these problems and provide the first unified risk analysis of learning with random Fourier features using the squared error and Lipschitz continuous loss functions. In our bounds, the trade-off between the computational cost and the expected risk convergence rate is problem specific and expressed in terms of the regularization parameter and the \emph{number of effective degrees of freedom}. We study both the standard random Fourier features method for which we improve the existing bounds on the number of features required to guarantee the corresponding minimax risk convergence rate of kernel ridge regression, as well as a data-dependent modification which samples features proportional to \emph{ridge leverage scores} and further reduces the required number of features. As ridge leverage scores are expensive to compute, we devise a simple approximation scheme which provably reduces the computational cost without loss of statistical efficiency.

Motivation & Objective

• To address the gap between pessimistic theoretical bounds and strong empirical performance in random Fourier features.
• To unify the theoretical analysis of random Fourier features across different learning tasks.
• To derive problem-specific trade-offs between computational cost and expected risk convergence rate.
• To improve bounds on the number of features needed for kernel ridge regression minimax risk convergence.
• To develop a data-dependent sampling method that reduces required features while preserving statistical efficiency.

Proposed method

• Proposes a unified risk analysis framework for random Fourier features using squared error and Lipschitz continuous loss functions.
• Expresses the trade-off between computational cost and risk convergence rate in terms of the regularization parameter and effective degrees of freedom.
• Analyzes the standard random Fourier features method and improves existing bounds on required feature counts for minimax risk convergence.
• Introduces a data-dependent variant that samples features proportional to ridge leverage scores to reduce feature count.
• Develops a provably efficient approximation scheme for ridge leverage scores to reduce computational cost without sacrificing statistical performance.
• Uses concentration inequalities and spectral analysis to derive generalization bounds under mild assumptions on the kernel and data distribution.

Experimental results

Research questions

• RQ1 How can a unified theoretical analysis of random Fourier features be developed across different learning tasks?
• RQ2 What is the precise trade-off between computational cost and expected risk convergence rate in random Fourier feature learning?
• RQ3 Can the number of required features be reduced while maintaining minimax risk convergence for kernel ridge regression?
• RQ4 How does data-dependent sampling based on ridge leverage scores improve feature efficiency compared to standard i.i.d. sampling?
• RQ5 Can an efficient approximation of ridge leverage scores be designed that maintains statistical performance while reducing computation?

Key findings

• The paper improves existing bounds on the number of random Fourier features required to achieve the minimax risk convergence rate for kernel ridge regression.
• The data-dependent sampling method based on ridge leverage scores reduces the required number of features compared to standard i.i.d. sampling.
• The proposed approximation scheme for ridge leverage scores provably reduces computational cost without loss of statistical efficiency.
• The unified risk analysis expresses the trade-off between computational cost and risk convergence in terms of the regularization parameter and effective degrees of freedom.
• Empirical results confirm that the data-dependent method achieves lower risk with fewer features, aligning with theoretical predictions.