[Paper Review] Distributed Mean Estimation with Limited Communication

Ananda Theertha Suresh, Felix X. Yu+2 more|arXiv (Cornell University)|Nov 2, 2016

Stochastic Gradient Optimization Techniques23 references148 citations

TL;DR

The paper develops communication-efficient, distribution-free protocols for distributed mean estimation, achieving minimax-optimal MSE versus communication cost and applying them to distributed Lloyd’s algorithm for k-means and power iteration for PCA.

ABSTRACT

Motivated by the need for distributed learning and optimization algorithms with low communication cost, we study communication efficient algorithms for distributed mean estimation. Unlike previous works, we make no probabilistic assumptions on the data. We first show that for $d$ dimensional data with $n$ clients, a naive stochastic binary rounding approach yields a mean squared error (MSE) of $Θ(d/n)$ and uses a constant number of bits per dimension per client. We then extend this naive algorithm in two ways: we show that applying a structured random rotation before quantization reduces the error to $\mathcal{O}((\log d)/n)$ and a better coding strategy further reduces the error to $\mathcal{O}(1/n)$ and uses a constant number of bits per dimension per client. We also show that the latter coding strategy is optimal up to a constant in the minimax sense i.e., it achieves the best MSE for a given communication cost. We finally demonstrate the practicality of our algorithms by applying them to distributed Lloyd's algorithm for k-means and power iteration for PCA.

Motivation & Objective

• Motivate distributed mean estimation with low communication cost without assuming data distributions.
• Study the minimax mean-squared error (MSE) under fixed communication budgets.
• Develop and compare multiple quantization and coding schemes to minimize MSE.
• Demonstrate practicality by applying algorithms to distributed Lloyd’s algorithm and PCA power iteration.

Proposed method

• Start with stochastic uniform quantization as a baseline.
• Introduce k-level stochastic quantization to reduce MSE.
• Apply stochastic rotated quantization to further reduce MSE via a random rotation before quantization.
• Use variable-length coding (arithmetic/Huffman) to compress quantized levels.
• Incorporate client sampling to trade off communication and MSE.
• Prove minimax lower bounds and show optimality up to constants.

Experimental results

Research questions

• RQ1 What is the best achievable MSE for distributed mean estimation under a given communication budget c, without distributional assumptions?
• RQ2 Can random rotations before quantization reduce MSE for fixed communication costs?
• RQ3 How do different quantization and coding strategies compare in terms of MSE and communication for distributed mean estimation?
• RQ4 Do the proposed schemes extend effectively to practical tasks like distributed Lloyd’s algorithm for k-means and power iteration for PCA?

Key findings

• A naive stochastic binary quantization yields MSE Theta(d/n) with constant bits per dimension.
• Stochastic k-level quantization improves to MSE O(d/(n(k-1)^2)) for fixed X i bounds, with communication n·(d log2 k + O(1)) bits.
• Stochastic rotated quantization with a random rotation achieves MSE O((log d)/n) up to constants for the same communication.
• Variable-length coding with arithmetic/Huffman coding yields MSE comparable to non-rotated quantization but with reduced communication, achieving O(d) bits per client for k <= sqrt(d).
• There exists a universal constant t<1 such that for c ≤ nd t, the minimax MSE is Theta(min(1, d/c)), establishing a linear scaling between communication and MSE with dimension.
• The schemes are demonstrated on distributed Lloyd’s algorithm (k-means) and power iteration (PCA).