[Paper Review] Collaborative Deep Learning in Fixed Topology Networks

Zhanhong Jiang, Aditya Balu+2 more|arXiv (Cornell University)|Jun 23, 2017

Distributed Control Multi-Agent Systems17 references77 citations

TL;DR

The paper introduces consensus-based distributed SGD (CDSGD) and its momentum variant (CDMSGD) for collaborative deep learning over fixed-topology graphs, enabling data parallelism and decentralized computation with convergence guarantees for convex and nonconvex objectives, plus empirical validation against centralized SGD and FedAvg on CIFAR-10/100.

ABSTRACT

There is significant recent interest to parallelize deep learning algorithms in order to handle the enormous growth in data and model sizes. While most advances focus on model parallelization and engaging multiple computing agents via using a central parameter server, aspect of data parallelization along with decentralized computation has not been explored sufficiently. In this context, this paper presents a new consensus-based distributed SGD (CDSGD) (and its momentum variant, CDMSGD) algorithm for collaborative deep learning over fixed topology networks that enables data parallelization as well as decentralized computation. Such a framework can be extremely useful for learning agents with access to only local/private data in a communication constrained environment. We analyze the convergence properties of the proposed algorithm with strongly convex and nonconvex objective functions with fixed and diminishing step sizes using concepts of Lyapunov function construction. We demonstrate the efficacy of our algorithms in comparison with the baseline centralized SGD and the recently proposed federated averaging algorithm (that also enables data parallelism) based on benchmark datasets such as MNIST, CIFAR-10 and CIFAR-100.

Motivation & Objective

• Motivate scalable distributed deep learning where each agent holds local private data and communication is constrained by a fixed topology.
• Develop CDSGD (and CDMSGD) to achieve data parallelism with decentralized computation under network constraints.
• Provide convergence analysis for strongly convex and nonconvex objectives using Lyapunov function construction.
• Benchmark against centralized SGD and Federated Averaging to assess convergence rate, accuracy, and generalization.

Proposed method

• Define a distributed empirical risk minimization problem over a fixed undirected graph with a doubly stochastic interaction matrix Pi.
• Propose CDSGD: x_k+1^j = sum_{l in Nb(j)} pi_jl x_k^l − alpha g_j(x_k^j) for each agent j.
• Introduce Lyapunov function V(x, alpha) = (N/n) 1^T F(x) + (1/(2 alpha)) ||x||_{I−Pi}^2 to analyze convergence.
• Establish consensus results showing E[||x_k^j − s_k||] ≤ alpha L / (1 − lambda_2(Pi)).
• Provide convergence theorems for strongly convex (linear convergence to a neighborhood) and nonconvex (bounded gradient sums) cases under Assumptions 1–3.
• Mention extensions to momentum variants (CDMSGD) and diminishing step sizes (supplementary material).

Experimental results

Research questions

• RQ1 Can CDSGD achieve consensus and converge when data is distributed across agents connected by a fixed topology?
• RQ2 What convergence rates and steady-state accuracies are attainable for strongly convex and nonconvex objectives under CDSGD/CDMSGD?
• RQ3 How does network topology (spectral gap) affect convergence, consensus, and final accuracy compared to centralized SGD or FedAvg?
• RQ4 Do CDSGD/CDMSGD improve generalization (training vs validation gap) compared to centralized baselines and FedAvg?
• RQ5 How do fixed vs diminishing step sizes influence convergence regimes and practical performance?

Key findings

• CDSGD achieves consensus across agents with a finite step size, with the consensus error bounded by alpha and graph spectral gap (lambda_2 of Pi).
• For strongly convex objectives, CDSGD converges linearly to a neighborhood of the optimum; the neighborhood shrinks with smaller step sizes and improves with larger spectral gap.
• For nonconvex objectives, CDSGD yields bounded sum of gradient norms over iterations, implying convergence to stationary points in a practical sense.
• CDMSGD can outperform FedAvg in steady-state accuracy while retaining decentralized computation, approaching centralized SGD performance given enough epochs.
• Empirical results on CIFAR-10/100 show CDSGD attaining comparable or higher final accuracy than centralized SGD or FedAvg, with smaller generalization gaps; network size and topology positively influence consensus stability and learning dynamics.