[Paper Review] FTPipeHD: A Fault-Tolerant Pipeline-Parallel Distributed Training Framework for Heterogeneous Edge Devices

Yuhao Chen, Qianqian Yang+3 more|arXiv (Cornell University)|Oct 6, 2021

IoT and Edge/Fog Computing39 references4 citations

TL;DR

FTPipeHD is a fault-tolerant, pipeline-parallel distributed training framework for heterogeneous edge devices that dynamically optimizes model partitioning based on real-time device capacity and uses periodic weight redistribution to enable rapid recovery from device failures. It achieves 6.8x faster training than state-of-the-art methods under high heterogeneity and maintains performance even during failures.

ABSTRACT

With the increased penetration and proliferation of Internet of Things (IoT) devices, there is a growing trend towards distributing the power of deep learning (DL) across edge devices rather than centralizing it in the cloud. This development enables better privacy preservation, real-time responses, and user-specific models. To deploy deep and complex models to edge devices with limited resources, model partitioning of deep neural networks (DNN) model is necessary, and has been widely studied. However, most of the existing literature only considers distributing the inference model while still relying centralized cloud infrastructure to generate this model through training. In this paper, we propose FTPipeHD, a novel DNN training framework that trains DNN models across distributed heterogeneous devices with fault tolerance mechanism. To accelerate the training with time-varying computing power of each device, we optimize the partition points dynamically according to real-time computing capacities. We also propose a novel weight redistribution approach that replicates the weights to both the neighboring nodes and the central node periodically, which combats the failure of multiple devices during training while incurring limited communication cost. Our numerical results demonstrate that FTPipeHD is 6.8x faster in training than the state of the art method when the computing capacity of the best device is 10x greater than the worst one. It is also shown that the proposed method is able to accelerate the training even with the existence of device failures.

Motivation & Objective

• To address the limitations of centralized cloud training, which compromises privacy, latency, and personalization in IoT applications.
• To enable on-device training of deep neural networks on resource-constrained edge devices by distributing training across multiple heterogeneous devices.
• To overcome the challenges of dynamic device availability and variable computing capacity in edge environments.
• To provide fault tolerance with minimal communication overhead during device failures in distributed training.
• To accelerate training through dynamic model partitioning and efficient weight replication across neighboring and central nodes.

Proposed method

• Dynamically adjusts model partition points in real time based on the current computing capacity of each edge device to balance workload and accelerate training.
• Introduces a novel weight redistribution mechanism that replicates model weights to both adjacent workers and a central node at regular intervals.
• Uses pipeline parallelism to distribute forward and backward passes across multiple devices, enabling concurrent computation across stages.
• Employs chain replication and global replication to ensure fault tolerance: if a device fails, neighboring and central nodes can immediately take over its workload.
• Minimizes communication cost by limiting weight replication to periodic intervals rather than per-batch transmission.
• Supports continuous learning by enabling incremental model updates using new data on edge devices without overfitting.

Experimental results

Research questions

• RQ1 How can distributed DNN training be accelerated on heterogeneous edge devices with varying and time-varying computing capacities?
• RQ2 What mechanisms can ensure fault tolerance in pipeline-parallel training without incurring prohibitive communication costs?
• RQ3 How can on-device training preserve privacy and enable real-time inference while maintaining model performance?
• RQ4 To what extent can dynamic model partitioning improve training speed under high device heterogeneity?
• RQ5 How effective is the proposed weight redistribution scheme in recovering from device failures with minimal performance degradation?

Key findings

• FTPipeHD achieves 6.8x faster training compared to the state-of-the-art method when the best device is 10x faster than the worst device.
• After a device failure, FTPipeHD recovers and maintains training performance with only a 2.24-second recovery overhead, while ResPipe takes 59.18 minutes to train one epoch post-failure.
• The training time per batch remains nearly unchanged after fault recovery in FTPipeHD, whereas ResPipe's training time increases significantly and stays high.
• The proposed fault tolerance mechanism reduces post-failure training time by 6.9x compared to ResPipe, making the recovery cost negligible in practice.
• On three Raspberry Pi devices, FTPipeHD successfully enables continuous learning with a 10% data stream, restoring model accuracy to pre-fine-tuning levels over time.
• Single-device training fails after 499 batches due to memory constraints, validating the necessity of distributed training on edge devices.