[Paper Review] Communication Complexity of Distributed Convex Learning and Optimization

Yossi Arjevani, Ohad Shamir|arXiv (Cornell University)|Jun 5, 2015

Stochastic Gradient Optimization Techniques23 references82 citations

TL;DR

This paper establishes fundamental communication complexity lower bounds for distributed convex optimization, showing that even with unlimited local computation, many communication rounds are necessary when local functions are unrelated. It proves tight bounds matching existing accelerated methods, and identifies conditions under which communication can be significantly reduced when local functions are statistically similar.

ABSTRACT

We study the fundamental limits to communication-efficient distributed methods for convex learning and optimization, under different assumptions on the information available to individual machines, and the types of functions considered. We identify cases where existing algorithms are already worst-case optimal, as well as cases where room for further improvement is still possible. Among other things, our results indicate that without similarity between the local objective functions (due to statistical data similarity or otherwise) many communication rounds may be required, even if the machines have unbounded computational power.

Motivation & Objective

• To identify fundamental limits on communication efficiency in distributed convex optimization under various assumptions.
• To determine whether existing distributed algorithms are optimal in the worst-case scenario.
• To analyze how data similarity across machines affects communication complexity.
• To derive tight lower bounds on the number of communication rounds required to achieve a given accuracy.
• To explore the role of smoothness, strong convexity, and structural assumptions on algorithmic performance.

Proposed method

• Derives communication complexity lower bounds using information-theoretic techniques, particularly mutual information and Pinsker’s inequality.
• Introduces a parameter δ to quantify similarity between local objective functions, enabling unified analysis of related and unrelated cases.
• Applies a structural assumption on algorithms to ensure the lower bounds are meaningful and generalizable.
• Uses a randomized matrix construction with symmetric properties to create hard instances for lower bound analysis.
• Analyzes the mutual information between transmitted messages and local function parameters to bound algorithmic accuracy.
• Combines acceleration techniques with Moreau proximal smoothing to suggest potential optimal algorithms for non-smooth cases.

Experimental results

Research questions

• RQ1 What is the minimum number of communication rounds required to solve distributed convex optimization when local functions are unrelated?
• RQ2 Can existing distributed algorithms be improved in terms of communication efficiency, or are they already worst-case optimal?
• RQ3 How does statistical similarity between local data (quantified by δ) affect the communication complexity of distributed optimization?
• RQ4 Are accelerated gradient methods optimal in the communication complexity sense for smooth and strongly convex functions?
• RQ5 What are the fundamental limits of communication efficiency when local functions are non-smooth or non-strongly convex?

Key findings

• For smooth and λ-strongly convex functions with unrelated local objectives, the communication complexity is Ω(√(1/λ) log(1/ε)), which is matched by accelerated gradient descent.
• For smooth convex functions with unrelated objectives, the lower bound is Ω(√(1/ε)), which is tight and matched by accelerated methods.
• For non-smooth λ-strongly convex functions, the lower bound is Ω(√(1/(λε))), suggesting potential for optimal algorithms combining acceleration and proximal smoothing.
• For general convex non-smooth functions, the lower bound is Ω(1/ε), indicating that high accuracy requires many communication rounds.
• When local functions are related (δ-related), the communication complexity drops to Ω(√(δ/λ) log(1/ε)), and this bound is matched (up to constants) by the DISCO algorithm for quadratic functions.
• Even with unbounded local computation, communication complexity cannot be reduced below these lower bounds in the unrelated case, showing fundamental limits.