[Paper Review] Communication-Efficient Algorithms For Distributed Optimization

Mota, João F. C.|arXiv (Cornell University)|Dec 1, 2013

Distributed Control Multi-Agent Systems208 references4 citations

TL;DR

This paper proposes communication-efficient distributed optimization algorithms for networks where each node holds a local cost function and collaborates to minimize their sum. By exploiting sparsity in function dependencies through a novel classification scheme, the authors design ADMM-based algorithms that reduce communication rounds—demonstrating superior performance over prior methods in consensus, SVMs, compressed sensing, and model predictive control.

ABSTRACT

This thesis is concerned with the design of distributed algorithms for solving optimization problems. We consider networks where each node has exclusive access to a cost function, and design algorithms that make all nodes cooperate to find the minimum of the sum of all the cost functions. Several problems in signal processing, control, and machine learning can be posed as such optimization problems. Given that communication is often the most energy-consuming operation in networks, it is important to design communication-efficient algorithms. The main contributions of this thesis are a classification scheme for distributed optimization and a set of corresponding communication-efficient algorithms. The class of optimization problems we consider is quite general, since each function may depend on arbitrary components of the optimization variable, and not necessarily on all of them. In doing so, we go beyond the common assumption in distributed optimization and create additional structure that can be used to reduce the number of communications. This structure is captured by our classification scheme, which identifies easier instances of the problem, for example the standard distributed optimization problem, where all functions depend on all the components of the variable. In our algorithms, no central node coordinates the network, all the communications occur between neighboring nodes, and the data associated with each node is processed locally. We show several applications including average consensus, support vector machines, network flows, and several distributed scenarios for compressed sensing. We also propose a new framework for distributed model predictive control. Through extensive numerical experiments, we show that our algorithms outperform prior distributed algorithms in terms of communication-efficiency, even some that were specifically designed for a particular application.

Motivation & Objective

• Address the high communication cost in distributed optimization by minimizing the number of inter-node messages.
• Develop a general framework for distributed optimization where each function depends only on a subset of variables, enabling structural exploitation.
• Design communication-efficient algorithms that maintain full distribution (no central coordination) and local processing at each node.
• Provide a classification scheme for distributed optimization problems based on variable dependency structure to identify easier-to-solve instances.
• Demonstrate the approach across diverse applications including average consensus, support vector machines, network flows, and compressed sensing.

Proposed method

• Propose a reformulation of the distributed optimization problem using block-structured variables and consensus constraints.
• Apply the 2-block ADMM to the reformulated problem, enabling distributed updates via alternating minimization over primal and dual variables.
• Leverage variable sparsity (i.e., each function depends only on a subset of components) to decompose subproblems and reduce communication.
• Derive closed-form updates for dual variables by exploiting symmetry and pairwise constraints, simplifying the dual update to depend only on local and neighbor estimates.
• Introduce a new dual variable aggregation γ(p),k_l = ∑_{j∈Np∩Vl} λpj,k_l to reduce communication overhead and enable efficient local updates.
• Use the identity z{p,j},k+1_l = (x(p),k+1_l + x(j),k+1_l)/2 to simplify consensus constraints and reduce the number of required messages.

Experimental results

Research questions

• RQ1 How can the dependency structure of local cost functions be exploited to reduce communication in distributed optimization?
• RQ2 Can a classification scheme for distributed optimization problems be designed based on variable sparsity, and does it lead to more efficient algorithms?
• RQ3 To what extent can ADMM-based algorithms be simplified and made communication-efficient when functions depend only on subsets of variables?
• RQ4 How does the proposed algorithm compare to state-of-the-art communication-efficient methods in terms of convergence speed and message count?
• RQ5 Can the framework be applied to real-world problems like support vector machines, compressed sensing, and distributed model predictive control with improved efficiency?

Key findings

• The proposed algorithm requires fewer communication rounds than existing distributed optimization methods, including those specifically designed for individual applications.
• The classification scheme identifies simpler problem instances—such as fully coupled problems—enabling tailored, more efficient algorithms.
• The use of symmetric dual variable initialization ensures that the dual update simplifies to z{p,j},k+1_l = (x(p),k+1_l + x(j),k+1_l)/2, reducing communication complexity.
• The algorithm achieves convergence with significantly reduced message passing, as validated through extensive numerical experiments on average consensus and support vector machines.
• The framework enables efficient distributed model predictive control by reformulating the problem within the proposed optimization structure.
• The derived update for the aggregated dual variable γ(p),k+1_l = γ(p),k_l + (ρ/2) ∑_{j∈Np∩Vl} (x(p),k+1_l - x(j),k+1_l) enables local computation and reduces reliance on global coordination.