[Paper Review] Distributed Machine Learning through Heterogeneous Edge Systems

Hanpeng Hu, Dan Wang+1 more|arXiv (Cornell University)|Nov 16, 2019

Stochastic Gradient Optimization Techniques24 references4 citations

TL;DR

This paper proposes ADSP, a novel parameter synchronization model for distributed machine learning in heterogeneous edge systems that eliminates waiting time by allowing faster workers to continuously train while committing updates at strategically timed intervals. ADSP ensures convergence and accelerates training by dynamically adjusting commit rates using a momentum-based online search, achieving up to 62.4% faster convergence than state-of-the-art methods.

ABSTRACT

Many emerging AI applications request distributed machine learning (ML) among edge systems (e.g., IoT devices and PCs at the edge of the Internet), where data cannot be uploaded to a central venue for model training, due to their large volumes and/or security/privacy concerns. Edge devices are intrinsically heterogeneous in computing capacity, posing significant challenges to parameter synchronization for parallel training with the parameter server (PS) architecture. This paper proposes ADSP, a parameter synchronization scheme for distributed machine learning (ML) with heterogeneous edge systems. Eliminating the significant waiting time occurring with existing parameter synchronization models, the core idea of ADSP is to let faster edge devices continue training, while committing their model updates at strategically decided intervals. We design algorithms that decide time points for each worker to commit its model update, and ensure not only global model convergence but also faster convergence. Our testbed implementation and experiments show that ADSP outperforms existing parameter synchronization models significantly in terms of ML model convergence time, scalability and adaptability to large heterogeneity.

Motivation & Objective

• To address the challenge of inefficient training in heterogeneous edge systems where workers vary significantly in computing and communication capacity.
• To minimize waiting time in parameter synchronization, which dominates training time in existing models.
• To ensure global model convergence while maximizing utilization of faster edge devices.
• To design a scalable and adaptive synchronization model that performs well under high heterogeneity and network latency.
• To outperform existing parameter synchronization models in convergence speed, scalability, and adaptability to real-world edge environments.

Proposed method

• ADSP introduces a no-waiting strategy where faster workers continue training without waiting for slower ones.
• It enforces all workers to commit model updates at synchronized, strategically chosen intervals to maintain convergence.
• A momentum-based online search algorithm computes optimal global commit rates and dynamically adjusts individual worker commit frequencies.
• The model treats total processing capacity as the sum of local computation time and communication delay per commit, enabling it to handle both compute and communication heterogeneity.
• ADSP ensures that each worker contributes approximately equal numbers of commits over time, regardless of speed differences.
• The system is implemented on a real testbed using diverse edge devices and evaluated on real-world AI workloads.

Experimental results

Research questions

• RQ1 How can parameter synchronization in distributed edge learning be optimized to minimize waiting time caused by worker heterogeneity?
• RQ2 Can a synchronization model be designed to ensure global model convergence while allowing faster workers to train continuously?
• RQ3 What is the optimal commit rate strategy that balances convergence speed and stability in heterogeneous environments?
• RQ4 How does the model perform under varying degrees of computational and communication heterogeneity?
• RQ5 To what extent does the model scale with increasing numbers of edge workers in real-world deployments?

Key findings

• ADSP achieves up to 62.4% faster convergence time compared to Fixed ADACOMM when the heterogeneity degree H = 3.2.
• The performance gap between ADSP and Fixed ADACOMM widens with increasing heterogeneity, demonstrating ADSP’s superior adaptability to diverse edge devices.
• ADSP maintains fast convergence even at scale, with 36 workers, outperforming Fixed ADACOMM significantly as system size increases.
• ADSP is robust to network latency: its convergence speedup over baselines increases with higher communication delays, due to reduced sensitivity to communication overhead.
• ADSP’s no-waiting strategy reduces idle time, enabling faster utilization of high-capacity edge devices and improving overall system efficiency.
• The model effectively handles both computational and communication heterogeneity by treating total processing capacity as a unified metric, making it resilient to diverse edge conditions.