[Paper Review] Distributed Deep Learning in Open Collaborations

M.G. Diskin, Alexey Bukhtiyarov+14 more|arXiv (Cornell University)|Jun 18, 2021

Scientific Computing and Data Management116 references4 citations

TL;DR

This paper proposes DeDLOC, a novel algorithmic framework for distributed deep learning in open, volunteer-based collaborations using heterogeneous, unreliable hardware. By dynamically adapting to varying compute and network conditions, DeDLOC enables efficient training comparable to traditional setups—demonstrated by successfully pretraining a large language model (sahajBERT-XL) in 22 days with 14 volunteers using hybrid GPU and TPU resources at a fraction of the cost.

ABSTRACT

Modern deep learning applications require increasingly more compute to train state-of-the-art models. To address this demand, large corporations and institutions use dedicated High-Performance Computing clusters, whose construction and maintenance are both environmentally costly and well beyond the budget of most organizations. As a result, some research directions become the exclusive domain of a few large industrial and even fewer academic actors. To alleviate this disparity, smaller groups may pool their computational resources and run collaborative experiments that benefit all participants. This paradigm, known as grid- or volunteer computing, has seen successful applications in numerous scientific areas. However, using this approach for machine learning is difficult due to high latency, asymmetric bandwidth, and several challenges unique to volunteer computing. In this work, we carefully analyze these constraints and propose a novel algorithmic framework designed specifically for collaborative training. We demonstrate the effectiveness of our approach for SwAV and ALBERT pretraining in realistic conditions and achieve performance comparable to traditional setups at a fraction of the cost. Finally, we provide a detailed report of successful collaborative language model pretraining with 40 participants.

Motivation & Objective

• To address the growing computational cost of training state-of-the-art deep learning models, which limits access to only large institutions and corporations.
• To overcome the challenges of training in volunteer computing environments—such as variable bandwidth, latency, and unreliable connectivity—commonly unsuitable for standard distributed training methods.
• To enable collaborative, decentralized training of large models using underutilized, heterogeneous devices (e.g., consumer GPUs, TPUs, workstations) without requiring dedicated infrastructure.
• To reduce the environmental and financial costs of large-scale model training by maximizing utilization of existing, idle computational resources.
• To demonstrate the feasibility and efficiency of open, community-driven pretraining of large language models through real-world deployment.

Proposed method

• DeDLOC dynamically adapts its training strategy based on real-time hardware and network conditions, combining elements of parameter server, All-Reduce, and decentralized SGD.
• The framework uses a hybrid communication strategy that selects optimal synchronization patterns (e.g., All-Reduce, parameter server, or intermediate) depending on device capabilities and network latency.
• It supports mixed-precision training across heterogeneous accelerators (e.g., float16 on GPUs, bfloat16 on TPUs) by aligning precision formats to prevent divergence.
• The system employs a decentralized, fault-tolerant training pipeline that handles dynamic participant joining and leaving, with automatic recovery from disconnections.
• It leverages model checkpointing and gradient aggregation techniques to maintain training stability and throughput in volatile environments.
• The architecture is designed to be lightweight and modular, enabling integration with existing deep learning frameworks like Hugging Face Transformers.

Experimental results

Research questions

• RQ1 Can distributed deep learning be effectively scaled in open, volunteer-based collaborations with highly heterogeneous and unreliable hardware?
• RQ2 How can training efficiency be maintained in environments with variable bandwidth, high latency, and frequent participant churn?
• RQ3 What algorithmic strategies can dynamically adapt to mixed-precision and mixed-architecture hardware (e.g., GPUs and TPUs) to ensure convergence and performance?
• RQ4 To what extent can collaborative training reduce the cost and environmental impact of pretraining large language models?
• RQ5 Can a community of independent contributors successfully train a large-scale language model (e.g., sahajBERT-XL) without centralized infrastructure?

Key findings

• The DeDLOC framework successfully trained sahajBERT-XL in 22 days using 14 volunteers, achieving performance that outperforms both baseline models and the original sahajBERT on downstream tasks.
• The collaborative training setup achieved results comparable to traditional high-end GPU clusters, but at a significantly reduced cost and with more efficient hardware utilization.
• Despite differences in mixed-precision support (float16 on GPUs, bfloat16 on TPUs), the framework prevented training divergence by aligning computation to float32 on TPUs while maintaining float16 on GPUs.
• The TPUv3-8 instance outperformed any single GPU node, demonstrating the effectiveness of heterogeneous accelerator utilization in the framework.
• The system maintained stable training throughput and convergence even with dynamic participant availability and variable network conditions.
• The environmental impact was reduced by leveraging existing, underused hardware and minimizing the need for large-scale cooling and energy-intensive data centers.