[Paper Review] SRP: Efficient class-aware embedding learning for large-scale data via supervised random projections

Amir-Hossein Karimi, Alexander Wong+1 more|arXiv (Cornell University)|Nov 7, 2018

Face and Expression Recognition4 citations

TL;DR

This paper proposes Supervised Random Projections (SRP), a novel method for efficient class-aware embedding learning in large-scale datasets by directly optimizing label dependence using kernel approximation and the Hilbert-Schmidt Independence Criterion (HSIC), bypassing computationally expensive SVD. SRP achieves performance competitive with SPCA and KSPCA while delivering 1–2 orders of magnitude faster computation, enabling scalable supervised dimensionality reduction for large data.

ABSTRACT

Supervised dimensionality reduction strategies have been of great interest. However, current supervised dimensionality reduction approaches are difficult to scale for situations characterized by large datasets given the high computational complexities associated with such methods. While stochastic approximation strategies have been explored for unsupervised dimensionality reduction to tackle this challenge, such approaches are not well-suited for accelerating computational speed for supervised dimensionality reduction. Motivated to tackle this challenge, in this study we explore a novel direction of directly learning optimal class-aware embeddings in a supervised manner via the notion of supervised random projections (SRP). The key idea behind SRP is that, rather than performing spectral decomposition (or approximations thereof) which are computationally prohibitive for large-scale data, we instead perform a direct decomposition by leveraging kernel approximation theory and the symmetry of the Hilbert-Schmidt Independence Criterion (HSIC) measure of dependence between the embedded data and the labels. Experimental results on five different synthetic and real-world datasets demonstrate that the proposed SRP strategy for class-aware embedding learning can be very promising in producing embeddings that are highly competitive with existing supervised dimensionality reduction methods (e.g., SPCA and KSPCA) while achieving 1-2 orders of magnitude better computational performance. As such, such an efficient approach to learning embeddings for dimensionality reduction can be a powerful tool for large-scale data analysis and visualization.

Motivation & Objective

• To address the high computational cost of supervised dimensionality reduction methods like SPCA and KSPCA on large-scale datasets.
• To develop a scalable alternative to SVD-based methods that avoids the O(dn²) complexity of full matrix decomposition.
• To leverage kernel approximation and HSIC symmetry to directly learn optimal class-aware embeddings without iterative optimization.
• To achieve competitive classification accuracy with significantly reduced wall-clock time for large datasets.
• To enable efficient visualization and downstream processing of high-dimensional data using a fast, closed-form embedding method.

Proposed method

• SRP formulates supervised embedding learning as a direct optimization of HSIC, a measure of dependence between projected data and labels, exploiting its symmetry for efficient computation.
• The method replaces spectral decomposition with a randomized projection scheme based on kernel approximation, avoiding SVD entirely.
• It uses random projection matrices to map data into a lower-dimensional space where dependence on labels is maximized, guided by the HSIC criterion.
• The approach is grounded in kernel approximation theory, allowing efficient computation of kernel-based dependencies without explicitly computing the full kernel matrix.
• For nonlinear cases, Kernel SRP (KSRP) extends SRP using kernel approximation techniques to handle nonlinear label dependencies.
• The method derives theoretical convergence bounds and provides time complexity guarantees, showing O(dn²) for SRP and O(kₓn² + kᵧn²) for KSRP, significantly faster than KSPCA's O(n³).

Experimental results

Research questions

• RQ1 Can a randomized, SVD-free approach achieve comparable classification performance to SPCA and KSPCA while drastically reducing computational cost?
• RQ2 How does the use of kernel approximation and HSIC symmetry enable direct optimization of label-dependent embeddings without spectral decomposition?
• RQ3 What is the trade-off between embedding quality and computational efficiency when using supervised random projections on large-scale datasets?
• RQ4 How do SRP and KSRP compare to SPCA and KSPCA in terms of 1-NN classification accuracy and wall-clock execution time?
• RQ5 Can the proposed method scale to large datasets where KSPCA fails due to memory and time constraints?

Key findings

• SRP achieves 1–2 orders of magnitude faster computation than SPCA and KSPCA, with wall-clock time improvements of 5–20× in experiments.
• On all five datasets tested, SRP and KSRP achieved 1-NN classification accuracy within 2% of their exact counterparts (SPCA and KSPCA), demonstrating strong performance retention.
• For the MNIST dataset, KSPCA failed due to memory limits (64GB), while SRP remained efficient and effective, highlighting scalability advantages.
• Increasing the number of random projection bases (k) improved 1-NN performance, confirming that higher k preserves more discriminative information.
• The time complexity of SRP is O(dn²), and KSRP is O(max{kₓn², kᵧn²}), both significantly lower than KSPCA’s O(n³), aligning with observed runtime improvements.
• The proposed method enables effective 2D visualization of complex data (e.g., XOR and MNIST) with performance close to exact SPCA, demonstrating utility in data exploration.