[Paper Review] Semi-Dynamic Load Balancing: Efficient Distributed Learning in Non-Dedicated Environments

Chen Chen, Qizhen Weng+3 more|arXiv (Cornell University)|Jun 7, 2018

Stochastic Gradient Optimization Techniques68 references4 citations

TL;DR

This paper proposes LB-BSP, a semi-dynamic load balancing framework that reduces straggler effects in distributed machine learning by adjusting sample batch sizes at iteration boundaries based on real-time worker performance. It achieves up to 54% faster training by using NARX-based prediction for CPU clusters and adaptive batch sizing for GPU clusters, enabling efficient, non-intrusive load balancing in non-dedicated environments.

ABSTRACT

Machine learning (ML) models are increasingly trained in clusters with non-dedicated workers possessing heterogeneous resources. In such scenarios, model training efficiency can be negatively affected by stragglers -- workers that run much slower than others. Efficient model training requires eliminating such stragglers, yet for modern ML workloads, existing load balancing strategies are inefficient and even infeasible. In this paper, we propose a novel strategy called semi-dynamic load balancing to eliminate stragglers of distributed ML workloads. The key insight is that ML workers shall be load-balanced at iteration boundaries, being non-intrusive to intra-iteration execution. We develop LB-BSP based on such an insight, which is an integrated worker coordination mechanism that adapts workers' load to their instantaneous processing capabilities by right-sizing the sample batches at the synchronization barriers. We have custom-designed the batch sizing algorithm respectively for CPU and GPU clusters based on their own characteristics. LB-BSP has been implemented as a Python module for ML frameworks like TensorFlow and PyTorch. Our EC2 deployment confirms that LB-BSP is practical, effective and light-weight, and is able to accelerating distributed training by up to $54\%$.

Motivation & Objective

• To address straggler problems in distributed machine learning training on non-dedicated, heterogeneous clusters with variable resource availability.
• To design a load balancing strategy that avoids the inefficiencies of static and dynamic load balancing in iterative, tensor-based ML workloads.
• To enable efficient, non-intrusive load balancing by performing adjustments only at iteration boundaries, preserving intra-iteration execution efficiency.
• To develop platform-specific batch sizing strategies for CPU and GPU clusters based on real-time processing capability prediction.
• To evaluate the practicality, effectiveness, and low overhead of LB-BSP in real-world cloud environments like Amazon EC2.

Proposed method

• LB-BSP integrates with the Bulk Synchronous Parallel (BSP) model by introducing load balancing at synchronization barriers, adjusting batch sizes per worker to equalize processing times.
• For CPU clusters, a NARX recurrent neural network predicts worker processing speed using real-time CPU and memory resource metrics to estimate batch processing time.
• The batch size for each worker is dynamically set at iteration start based on predicted processing capability to balance workload across workers.
• For GPU clusters, a separate batch sizing algorithm is designed to account for the characteristics of GPU memory and compute throughput.
• The system uses non-blocking RPC communication between the BatchSizeManager and workers to minimize runtime overhead.
• Performance monitoring is done via execution logs (e.g., TensorFlow Timeline objects), with minimal impact on training throughput.

Experimental results

Research questions

• RQ1 How can load balancing be efficiently performed in distributed ML training without disrupting intra-iteration execution?
• RQ2 What is the most effective way to predict worker processing capability in dynamic, non-dedicated environments?
• RQ3 Can semi-dynamic load balancing based on batch size adjustment outperform traditional static and dynamic load balancing in ML workloads?
• RQ4 How does LB-BSP perform in terms of training speedup and system overhead across CPU and GPU clusters?
• RQ5 What is the impact of prediction accuracy on overall training efficiency in heterogeneous, shared environments?

Key findings

• LB-BSP achieves up to 54% faster distributed training in Amazon EC2 deployments by effectively eliminating stragglers in non-dedicated clusters.
• The NARX-based prediction model reduces RMSE by 40% compared to baseline methods like EMA and ARIMA, enabling more accurate batch size configuration.
• System overhead from LB-BSP is less than 1.1% of total iteration time even in clusters with 96 workers, confirming its lightweight nature.
• The framework maintains high performance on both CPU and GPU clusters, with tailored batch sizing strategies for each platform.
• Prediction robustness is demonstrated by NARX’s ability to suppress transient spikes while tracking deterministic performance improvements.
• LB-BSP outperforms existing load balancing approaches by avoiding work stealing and runtime migration, which are incompatible with tensor-based processing.