[Paper Review] Communication-efficient Distributed Sparse Linear Discriminant Analysis

Lu Tian, Quanquan Gu|arXiv (Cornell University)|Oct 15, 2016

Sparse and Compressive Sensing Techniques27 references15 citations

TL;DR

This paper proposes a communication-efficient distributed sparse linear discriminant analysis (LDA) method that splits high-dimensional data across m machines, computes local sparse LDA estimators, and aggregates them into a global estimator. The method achieves the same statistical rate as centralized estimation—O(√(s log d / N)) in ℓ2 norm—when m ≲ √(N / log d) / max(s, s′), and ensures model selection consistency under milder conditions than centralized methods.

ABSTRACT

We propose a communication-efficient distributed estimation method for sparse linear discriminant analysis (LDA) in the high dimensional regime. Our method distributes the data of size $N$ into $m$ machines, and estimates a local sparse LDA estimator on each machine using the data subset of size $N/m$. After the distributed estimation, our method aggregates the debiased local estimators from $m$ machines, and sparsifies the aggregated estimator. We show that the aggregated estimator attains the same statistical rate as the centralized estimation method, as long as the number of machines $m$ is chosen appropriately. Moreover, we prove that our method can attain the model selection consistency under a milder condition than the centralized method. Experiments on both synthetic and real datasets corroborate our theory.

Motivation & Objective

• Address the challenge of high-dimensional data in distributed machine learning by enabling efficient, scalable sparse LDA estimation.
• Reduce communication costs in distributed learning while maintaining statistical performance comparable to centralized methods.
• Achieve model selection consistency for sparse LDA estimators under weaker conditions than centralized approaches.
• Provide theoretical guarantees on estimation error and support recovery in high-dimensional, distributed settings.

Proposed method

• Distribute N total samples across m machines, each processing N/m samples to compute a local sparse LDA estimator via ℓ1-constrained optimization.
• Use debiased estimators on each worker machine to reduce bias and improve estimation accuracy.
• Aggregate local estimators at a master node by averaging, then sparsify the aggregated estimator to recover the true sparse structure.
• Employ a one-round communication protocol: each worker sends only a vector (not raw data) to the master, minimizing communication overhead.
• Theoretical analysis relies on sub-Gaussian and sub-Exponential concentration inequalities to bound estimation error in ℓ2 and other norms.
• Leverage restricted eigenvalue conditions on the sample covariance matrix to ensure consistency under high-dimensional asymptotics.

Experimental results

Research questions

• RQ1 Can a distributed sparse LDA method achieve the same statistical rate as centralized estimation while minimizing communication cost?
• RQ2 Under what conditions on the number of machines m does the distributed estimator maintain the optimal rate O(√(s log d / N)) in ℓ2 norm?
• RQ3 Does the proposed method achieve model selection consistency under weaker assumptions than centralized sparse LDA?
• RQ4 How does the communication efficiency of the one-round protocol compare to multi-round alternatives in terms of statistical performance?
• RQ5 What is the impact of data partitioning on estimation error and support recovery in high-dimensional sparse LDA?

Key findings

• The proposed distributed estimator achieves an ℓ2 estimation error bound of O(√(s log d / N) + max(s, s′)m√(s log d / N)), matching the centralized rate when m ≲ √(N / log d) / max(s, s′).
• The method attains model selection consistency under the condition minj |β∗j| ≳ √(log d / N), which is weaker than the condition required by the centralized method.
• The one-round communication protocol ensures communication efficiency, requiring only a single vector transmission per worker machine.
• Theoretical analysis confirms that the aggregated estimator maintains the same statistical rate as the centralized estimator, provided the number of machines is not too large relative to sample size and sparsity.
• The method is robust under high-dimensional asymptotics, with error bounds derived using sub-Gaussian and sub-Exponential concentration inequalities.
• Theoretical guarantees are established under the restricted eigenvalue condition on the sample covariance matrix, which holds with high probability when n ≳ s log d.