[Paper Review] Streaming Kernel Principal Component Analysis

Mina Ghashami, Daniel J. Perry+1 more|arXiv (Cornell University)|Dec 16, 2015

Sparse and Compressive Sensing Techniques3 citations

TL;DR

This paper proposes SKPCA, a streaming kernel principal component analysis method that maintains a small, logarithmic-space set of basis vectors using random feature maps and matrix sketching. It achieves spectral norm error bounds with improved dependence on the error parameter, outperforming state-of-the-art methods in test time while matching or exceeding them in accuracy and space efficiency.

ABSTRACT

Kernel principal component analysis (KPCA) provides a concise set of basis vectors which capture non-linear structures within large data sets, and is a central tool in data analysis and learning. To allow for non-linear relations, typically a full $n imes n$ kernel matrix is constructed over $n$ data points, but this requires too much space and time for large values of $n$. Techniques such as the Nyström method and random feature maps can help towards this goal, but they do not explicitly maintain the basis vectors in a stream and take more space than desired. We propose a new approach for streaming KPCA which maintains a small set of basis elements in a stream, requiring space only logarithmic in $n$, and also improves the dependence on the error parameter. Our technique combines together random feature maps with recent advances in matrix sketching, it has guaranteed spectral norm error bounds with respect to the original kernel matrix, and it compares favorably in practice to state-of-the-art approaches.

Motivation & Objective

• To address the high space and time complexity of traditional kernel PCA, which requires storing an n×n kernel matrix for large n.
• To develop a streaming algorithm that maintains basis vectors in real-time with minimal space, avoiding full kernel matrix computation.
• To improve the dependence on the error parameter ε in spectral and Frobenius norm error bounds.
• To achieve low test-time cost for mapping new data points to the kernel feature space.
• To combine random feature maps with matrix sketching for a provably accurate, efficient, and scalable streaming KPCA solution.

Proposed method

• The method uses random feature maps to embed the kernel-induced Reproducing Kernel Hilbert Space (RKHS) into a finite-dimensional Euclidean space.
• It applies matrix sketching techniques to maintain a low-rank approximation of the kernel matrix in a streaming fashion, reducing space to O(m log n) where m is the number of random features.
• The algorithm maintains a sketch of the data in the feature space and performs online SVD on the sketch to extract principal components.
• It uses a randomized dimensionality reduction approach to project data into a lower-dimensional space while preserving spectral norm error guarantees.
• The method supports incremental updates: new data points are mapped to the feature space and the sketch is updated without recomputing from scratch.
• The final representation is obtained by projecting the mapped data onto the top ℓ principal components, enabling efficient test-time inference.

Experimental results

Research questions

• RQ1 Can we design a streaming KPCA algorithm that maintains basis vectors with space complexity logarithmic in n, rather than O(n²)?
• RQ2 Can we achieve tighter dependence on the error parameter ε in spectral norm error bounds compared to existing methods?
• RQ3 Does combining random feature maps with matrix sketching yield better test-time performance than Nyström or random feature map-based approaches?
• RQ4 How does the proposed method compare in accuracy and efficiency across different data distributions and kernel types?
• RQ5 Can the method maintain provable error bounds while enabling real-time updates for large-scale streaming data?

Key findings

• SKPCA achieves a space complexity of O(m log n), where m is the number of random features, significantly reducing storage compared to O(n²) for full kernel matrix storage.
• The method provides guaranteed spectral norm error bounds of the form ||G - G'||₂ / n ≤ ε, with improved dependence on ε compared to prior approaches.
• For test-time inference, SKPCA requires only O(dm) time per data point, outperforming Nyström (O(cd + c²)) and RNCA (O(dm)) by orders of magnitude in some cases.
• In experiments on CPU, Adult, and Forest datasets, SKPCA achieved comparable or better Frobenius and spectral error than RNCA and Nyström, especially at small sample sizes.
• SKPCA demonstrated superior training time performance, with a significant advantage over RNCA due to avoiding O(n m²) outer product computations.
• The method showed robustness across different data distributions, with minimal error increase even when the signal-to-noise ratio was low.