[Paper Review] Asynchronous Decentralized Learning of a Neural Network

Xinyue Liang, Alireza M. Javid+2 more|arXiv (Cornell University)|Apr 10, 2020

Advanced Memory and Neural Computing27 references4 citations

TL;DR

This paper proposes asynchronous decentralized training of a self-size estimating feedforward neural network (dSSFN) using ARock-based ADMM, enabling centralized equivalence with reduced communication overhead. By allowing one-side activation and communication, asynchronous dSSFN achieves faster training—especially on sparse networks—while matching or outperforming synchronous counterparts in accuracy.

ABSTRACT

In this work, we exploit an asynchronous computing framework namely ARock to learn a deep neural network called self-size estimating feedforward neural network (SSFN) in a decentralized scenario. Using this algorithm namely asynchronous decentralized SSFN (dSSFN), we provide the centralized equivalent solution under certain technical assumptions. Asynchronous dSSFN relaxes the communication bottleneck by allowing one node activation and one side communication, which reduces the communication overhead significantly, consequently increasing the learning speed. We compare asynchronous dSSFN with traditional synchronous dSSFN in the experimental results, which shows the competitive performance of asynchronous dSSFN, especially when the communication network is sparse.

Motivation & Objective

• To design a decentralized neural network that achieves centralized equivalence in parameter learning without a master node.
• To reduce communication bottlenecks in decentralized training by enabling asynchronous, one-side communication.
• To develop a low-complexity, scalable learning framework suitable for heterogeneous nodes with varying processing speeds.
• To ensure convergence and performance parity with centralized training under non-convex optimization settings.
• To evaluate the impact of network sparsity and topology on learning efficiency and scalability.

Proposed method

• The method employs a layer-by-layer learning approach for a self-size estimating feedforward neural network (SSFN), where weight matrices are decomposed into a random matrix and an optimized submatrix.
• Convex optimization of the submatrix component is solved via ADMM, which ensures centralized equivalence and is compatible with decentralized and asynchronous computation.
• The ARock algorithm is used to implement asynchronous ADMM, allowing nodes to update using outdated information and communicate unilaterally, reducing synchronization delays.
• Each node performs local updates using its own data and exchanges only partial information with neighbors, minimizing communication overhead.
• The algorithm is applied iteratively across layers, with fixed step size η = 0.5 in all ADMM iterations.
• The framework supports both ring and circular network topologies, with performance evaluated under varying network degrees and node counts.

Experimental results

Research questions

• RQ1 Can asynchronous decentralized learning achieve centralized equivalence in a non-convex neural network training setup?
• RQ2 How does one-side activation and communication in ARock-based ADMM affect learning speed and convergence in decentralized neural networks?
• RQ3 What is the impact of network sparsity and topology on the training efficiency of decentralized deep learning models?
• RQ4 How does asynchronous dSSFN compare to synchronous dSSFN in terms of accuracy and training time across diverse datasets?
• RQ5 Can the proposed method maintain competitive performance while reducing communication overhead in decentralized systems?

Key findings

• Asynchronous dSSFN achieves centralized equivalence, with test accuracy matching or slightly exceeding that of centralized SSFN across all datasets.
• On the Vowel dataset, asynchronous dSSFN achieved 61.2% accuracy in 7.8 seconds, outperforming synchronous dSSFN (61.3% in 34.9 seconds) in training time.
• On the Caltech101 dataset, asynchronous dSSFN reached 75.4% accuracy in 27.4 seconds, compared to 74.2% in 427.5 seconds for synchronous dSSFN.
• In ring networks (d=2), asynchronous dSSFN training time increased only slightly with more nodes, while synchronous dSSFN showed monotonically increasing training time due to communication overhead.
• For sparse networks (low d), asynchronous dSSFN significantly outperforms synchronous dSSFN in training speed, demonstrating robustness to low connectivity.
• The method maintains competitive performance across diverse datasets including MNIST, NORB, Letter, and Satimage, with minimal accuracy degradation despite asynchronous updates.