[Paper Review] Federated Principal Component Analysis

Andreas Grammenos, Rodrigo Mendoza-Smith+2 more|arXiv (Cornell University)|Jul 18, 2019

Privacy-Preserving Technologies in Data55 references34 citations

TL;DR

Proposes a federated, asynchronous, and (b5, 6) -differentially private algorithm for PCA in memory-limited settings, with streaming local updates and a low-memory merging procedure.

ABSTRACT

We present a federated, asynchronous, and $(\varepsilon, δ)$-differentially private algorithm for PCA in the memory-limited setting. Our algorithm incrementally computes local model updates using a streaming procedure and adaptively estimates its $r$ leading principal components when only $\mathcal{O}(dr)$ memory is available with $d$ being the dimensionality of the data. We guarantee differential privacy via an input-perturbation scheme in which the covariance matrix of a dataset $\mathbf{X} \in \mathbb{R}^{d imes n}$ is perturbed with a non-symmetric random Gaussian matrix with variance in $\mathcal{O}\left(\left(\frac{d}{n} ight)^2 \log d ight)$, thus improving upon the state-of-the-art. Furthermore, contrary to previous federated or distributed algorithms for PCA, our algorithm is also invariant to permutations in the incoming data, which provides robustness against straggler or failed nodes. Numerical simulations show that, while using limited-memory, our algorithm exhibits performance that closely matches or outperforms traditional non-federated algorithms, and in the absence of communication latency, it exhibits attractive horizontal scalability.

Motivation & Objective

• Address privacy-preserving PCA on decentralized data without sharing raw data.
• Develop an asynchronous federated framework with memory-efficient local updates.
• Guarantee (b5, 6) -differential privacy via input perturbation in streaming covariance estimation.
• Provide a low-memory merging procedure to aggregate local subspaces while preserving privacy.
• Demonstrate robustness to data permutation and scalability in multi-node settings.

Proposed method

• Incrementally compute local PCA updates via streaming blocks on each client.
• Apply a differential privacy input-perturbation scheme to the covariance updates.
• Use a low-memory merging algorithm to combine subspaces from different clients.
• Extend MOD-SuLQ to streaming and non-symmetric noise for DP guarantees.
• Allow asynchronous, tree-structured federation so subspace updates are permutation-invariant.
• Dynamically adapt rank r using energy-based criteria to balance utility and DP constraints.

Experimental results

Research questions

• RQ1 How can PCA be computed in a federated, privacy-preserving, streaming setting with limited memory?
• RQ2 Can we securely merge local PCA updates without storing full datasets and while maintaining DP guarantees?
• RQ3 Does the framework achieve permutation-invariance and robustness to stragglers in a federated network?
• RQ4 What are the memory and communication trade-offs for streaming DP-PCA in decentralized networks?
• RQ5 How does the method perform compared to non-private PCA and other streaming PCA methods?

Key findings

• Federated-PCA achieves (b5, 6) -differential privacy with a variance-efficient non-symmetric noise scheme.
• The algorithm operates in an asynchronous, memory-{}low (O(d r)) setting with one pass per data column.
• A low-complexity merge procedure (Merge_r) privately aggregates incremental subspaces.
• Experiments show projection quality close to or exceeding traditional non-federated PCA under DP, with attractive horizontal scalability in absence of latency.
• FPCA-Edge adapts rank during streaming and remains robust to permutations of incoming data.
• Empirical results indicate favorable utility versus privacy when using MNIST and Wine datasets.