[Paper Review] Spectral approximations in machine learning

Darren Homrighausen, Daniel J. McDonald|arXiv (Cornell University)|Jul 21, 2011

Neural Networks and Applications12 references4 citations

TL;DR

This paper compares the Nyström extension and Gaussian projection methods for approximating eigenvectors of large kernel matrices in spectral machine learning. It finds that performance is task-dependent: Gaussian projection outperforms Nyström in classification with moderate sample size (m ≈ 10% of n), while Nyström excels in clustering and manifold reconstruction, with weighted sampling offering slight improvements over uniform sampling.

ABSTRACT

In many areas of machine learning, it becomes necessary to find the eigenvector decompositions of large matrices. We discuss two methods for reducing the computational burden of spectral decompositions: the more venerable Nystom extension and a newly introduced algorithm based on random projections. Previous work has centered on the ability to reconstruct the original matrix. We argue that a more interesting and relevant comparison is their relative performance in clustering and classification tasks using the approximate eigenvectors as features. We demonstrate that performance is task specific and depends on the rank of the approximation.

Motivation & Objective

• To provide a principled comparison of Nyström extension and Gaussian projection for spectral approximations in machine learning.
• To evaluate these methods not by matrix reconstruction accuracy, but by their impact on downstream learning tasks such as clustering, classification, and manifold recovery.
• To investigate the influence of approximation rank (m) and sampling strategy (uniform vs. weighted) on performance in real-world ML applications.
• To determine whether one method consistently outperforms the other across different learning tasks and data scales.

Proposed method

• The study uses the diffusion map from the graph Laplacian as the core spectral technique, requiring eigendecomposition of the normalized weight matrix W.
• The Nyström method approximates W's spectrum by sampling m columns (and rows) of W, then extending the eigendecomposition to the full matrix using interpolation.
• The Gaussian projection method uses random projections to embed the full matrix into a lower-dimensional subspace, then computes eigenvectors in this subspace and lifts them back.
• Two variants of Nyström are tested: uniform sampling and weighted sampling based on diagonal elements of W.
• The approximation quality is evaluated on three tasks: low-dimensional manifold recovery, clustering, and classification using SVM with 10-fold cross-validation.
• Performance is measured via test set misclassification rates and visual inspection of eigenvector embeddings, with true eigenvectors computed for small datasets (n=4800) as a benchmark.

Experimental results

Research questions

• RQ1 Does Gaussian projection outperform the Nyström method in downstream machine learning tasks such as classification and clustering?
• RQ2 How does the choice of sampling strategy (uniform vs. weighted) affect Nyström's performance in spectral approximations?
• RQ3 How does the approximation rank m influence the performance of both methods across different learning tasks?
• RQ4 Is matrix reconstruction accuracy a reliable proxy for downstream learning performance in spectral methods?
• RQ5 At what value of m does Gaussian projection begin to outperform Nyström in classification tasks?

Key findings

• For classification with m = 400 on a 4800-sample MNIST subset, Gaussian projection achieved 725 correct classifications (90.6% accuracy), outperforming Nyström methods (697–701 correct).
• In the same classification task, the true eigenvector method achieved 756 correct classifications (94.5% accuracy), showing that Gaussian projection incurs only a small efficiency loss.
• For manifold recovery, both Nyström and Gaussian projection successfully recover the underlying low-dimensional structure, though Gaussian projection's embedding is slightly more distorted than the true eigenvectors.
• In clustering, Gaussian projection produces a noisier embedding than both Nyström and the true eigenvectors, resulting in poorer separation of inner and outer rings.
• For larger m (e.g., m = 4050 on n=18,000), Gaussian projection's misclassification rate improves significantly and surpasses Nyström, especially when m ≥ 10% of n.
• Weighted Nyström slightly outperforms uniform Nyström in classification, but the difference is minimal, suggesting that sampling strategy has limited impact compared to the choice of approximation method.