[Paper Review] Communication Efficient Distributed Kernel Principal Component Analysis

Maria-Florina Balcan, Yingyu Liang+3 more|arXiv (Cornell University)|Mar 23, 2015

Face and Expression Recognition4 citations

TL;DR

This paper proposes a communication-efficient distributed kernel PCA algorithm that uses subspace embedding and adaptive sampling to compute global kernel principal components with relative error guarantees. It achieves near-optimal communication complexity of Õ(sρk/ε + sk²/ε³) words, independent of feature space dimension, and outperforms baselines in accuracy and communication efficiency on real-world datasets.

ABSTRACT

Kernel Principal Component Analysis (KPCA) is a key machine learning algorithm for extracting nonlinear features from data. In the presence of a large volume of high dimensional data collected in a distributed fashion, it becomes very costly to communicate all of this data to a single data center and then perform kernel PCA. Can we perform kernel PCA on the entire dataset in a distributed and communication efficient fashion while maintaining provable and strong guarantees in solution quality? In this paper, we give an affirmative answer to the question by developing a communication efficient algorithm to perform kernel PCA in the distributed setting. The algorithm is a clever combination of subspace embedding and adaptive sampling techniques, and we show that the algorithm can take as input an arbitrary configuration of distributed datasets, and compute a set of global kernel principal components with relative error guarantees independent of the dimension of the feature space or the total number of data points. In particular, computing $k$ principal components with relative error $ε$ over $s$ workers has communication cost $ ilde{O}(s ρk/ε+s k^2/ε^3)$ words, where $ρ$ is the average number of nonzero entries in each data point. Furthermore, we experimented the algorithm with large-scale real world datasets and showed that the algorithm produces a high quality kernel PCA solution while using significantly less communication than alternative approaches.

Motivation & Objective

• To address the high communication cost of transferring distributed, high-dimensional data to a central node for kernel PCA.
• To develop a distributed algorithm that maintains strong theoretical guarantees on solution quality for nonlinear feature extraction.
• To achieve communication complexity independent of the kernel feature space dimension, which can be infinite.
• To enable scalable, low-communication kernel PCA for large-scale applications such as image, text, and healthcare data analysis.
• To provide a practical and theoretically grounded solution for distributed spectral clustering and downstream machine learning tasks.

Proposed method

• The algorithm uses adaptive sampling to construct a small, representative subset of data points that preserves the global kernel subspace structure.
• It applies subspace embedding techniques to ensure the selected subset approximates the full data's kernel principal components within a (1+ε) relative error.
• The method operates in a master-worker architecture, where each worker holds a data partition and communicates only with the master.
• For polynomial and shift-invariant kernels (e.g., RBF), it achieves (1+ε)-approximation with additive error that can be made arbitrarily small.
• It introduces a novel distributed Column Subset Selection (CSS) algorithm as a subroutine, with communication cost O(sρk/ε + sk²), nearly matching the lower bound.
• The algorithm projects data onto the computed low-rank subspace, enabling downstream applications like distributed k-means clustering.

Experimental results

Research questions

• RQ1 Can we perform kernel PCA in a distributed setting with communication cost independent of the kernel feature space dimension?
• RQ2 Is it possible to achieve a (1+ε)-relative error approximation to the best rank-k kernel subspace using minimal communication?
• RQ3 How can we design a communication-efficient algorithm that selects a small, representative subset of data points to approximate the global kernel PCA solution?
• RQ4 Can the proposed method scale effectively across multiple workers while maintaining high solution quality?
• RQ5 Does the algorithm outperform uniform sampling and batch KPCA in terms of communication efficiency and approximation accuracy?

Key findings

• The algorithm achieves a communication cost of Õ(sρk/ε + sk²/ε³) words, which is nearly optimal and independent of the feature space dimension.
• On sparse datasets like bow and 20news, the algorithm leverages sparsity to achieve significantly lower error than uniform sampling methods.
• For the Gaussian RBF kernel, disKPCA achieves lower approximation error than uniform + batch KPCA even when the latter uses much more communication.
• On the higgs dataset, disKPCA achieves the same error as uniform + disLR with less than 1/5 of the communication cost.
• The algorithm enables effective distributed spectral clustering, outperforming baselines in the trade-off between communication and k-means objective value.
• Scaling experiments show a 2× speedup with 4× more workers, demonstrating favorable parallelization properties despite not being primarily optimized for computation.