[論文レビュー] Orthogonal Random Features

Felix X. Yu, Ananda Theertha Suresh+3 more|arXiv (Cornell University)|Oct 28, 2016

Generative Adversarial Networks and Image Synthesis被引用数 104

ひとこと要約

本論文は、Random Fourier Featuresにおいてランダムガウス行列を直交（および構造化）行列に置換することで、ガウスカーネルのカーネル推定誤差を低減し、構造化直交ランダム特徴量（SORF）を導入して精度と比較可能なまま計算を高速化することを示している。

ABSTRACT

We present an intriguing discovery related to Random Fourier Features: in Gaussian kernel approximation, replacing the random Gaussian matrix by a properly scaled random orthogonal matrix significantly decreases kernel approximation error. We call this technique Orthogonal Random Features (ORF), and provide theoretical and empirical justification for this behavior. Motivated by this discovery, we further propose Structured Orthogonal Random Features (SORF), which uses a class of structured discrete orthogonal matrices to speed up the computation. The method reduces the time cost from $\mathcal{O}(d^2)$ to $\mathcal{O}(d \log d)$, where $d$ is the data dimensionality, with almost no compromise in kernel approximation quality compared to ORF. Experiments on several datasets verify the effectiveness of ORF and SORF over the existing methods. We also provide discussions on using the same type of discrete orthogonal structure for a broader range of applications.

研究の動機と目的

• Motivate and analyze kernel approximation via random Fourier features for Gaussian kernels.
• Demonstrate that orthogonality in the projection matrix reduces kernel estimation error.
• Introduce Structured Orthogonal Random Features (SORF) to reduce computation from O(d^2) to O(d log d).
• Provide theoretical justification for orthogonal and structured projections and validate empirically across datasets.

提案手法

• Formulate ORF by replacing Gaussian random matrix G with S Q, where Q is a random orthogonal matrix and S is a diagonal scaling to match row norms.
• Prove ORF is an unbiased estimator of the Gaussian kernel and analyze its variance reduction compared to standard RFF.
• Introduce ORF′ as a simplified variant with W_ORF′ = sqrt(d)/σ Q and derive bias/variance guarantees.
• Propose SORF as W_SORF = (sqrt(d)/σ) H D1 H D2 H D3, where D_i are random diagonal sign matrices and H is the Walsh–Hadamard matrix, enabling O(D log d) computation and near-equivalent kernel quality.
• Discuss general applicability of the Hadamard-Diagonal structure beyond kernel approximation.

実験結果

リサーチクエスチョン

• RQ1 Does enforcing orthogonality on the random projection matrix improve Gaussian kernel approximation over standard Random Fourier Features?
• RQ2 Can a structured orthogonal transform (SORF) offer similar kernel quality with substantially reduced computational cost?
• RQ3 What are the bias and variance implications of ORF and SORF relative to RFF across varying data dimensionality and sample sizes?
• RQ4 Is the proposed structure generalizable to other kernel types and applications beyond kernel approximation?

主な発見

Dataset	D=2d	D=4d	D=6d	D=8d	D=10d	Exact
letter (d=16)	76.44 b1 1.04	81.61 b1 0.46	85.46 b1 0.56	86.58 b1 0.99	87.84 b1 0.59	90.10
forest (d=64)	77.61 b1 0.23	78.92 b1 0.30	79.29 b1 0.24	79.57 b1 0.21	79.85 b1 0.10	80.43
usps (d=256)	94.27 b1 0.38	94.98 b1 0.10	95.43 b1 0.22	95.66 b1 0.25	95.71 b1 0.18	95.57
cifar (d=512)	73.19 b1 0.23	75.06 b1 0.33	75.85 b1 0.30	76.28 b1 0.30	76.54 b1 0.31	78.71
mnist (d=1024)	94.83 b1 0.13	95.48 b1 0.10	95.85 b1 0.07	96.02 b1 0.06	95.98 b1 0.05	97.14
gisette (d=4096)	97.68 b1 0.28	97.74 b1 0.11	97.66 b1 0.25	97.70 b1 0.16	97.74 b1 0.05	97.60

• ORF provides an unbiased Gaussian kernel estimator with lower variance than RFF, especially for small z = ||x−y||/σ.
• SORF achieves near-identical kernel approximation quality to ORF while reducing time to O(D log d) and achieving minimal extra memory usage.
• On six datasets, ORF and SORF outperform RFF in kernel MSE for fixed D, with SORF matching ORF’s performance.
• Empirical results show ORF/SORF offer competitive or superior classification accuracy compared with RFF in SVM settings, with substantial speedups (e.g., up to 10x on gisette).
• The bias of ORF′ is small for large d, and its variance closely tracks ORF, supporting the practicality of simplified variants.