[Paper Review] EasyScale: Accuracy-consistent Elastic Training for Deep Learning

Mingzhen Li, Wencong Xiao+10 more|arXiv (Cornell University)|Aug 30, 2022

Stochastic Gradient Optimization Techniques4 citations

TL;DR

EasyScale is a novel elastic deep learning training system that maintains consistent model accuracy across varying GPU resources—both homogeneous and heterogeneous—by decoupling training behavior from resource allocation. It achieves this through a deterministic, accuracy-consistent training abstraction (EST) and efficient intra-/inter-job scheduling, improving cluster GPU utilization by 62.1% in production deployment.

ABSTRACT

Distributed synchronized GPU training is commonly used for deep learning. The resource constraint of using a fixed number of GPUs makes large-scale training jobs suffer from long queuing time for resource allocation, and lowers the cluster utilization. Adapting to resource elasticity can alleviate this but often introduces inconsistent model accuracy, due to lacking of capability to decouple model training procedure from resource allocation. We propose EasyScale, an elastic training system that achieves consistent model accuracy under resource elasticity for both homogeneous and heterogeneous GPUs. EasyScale preserves the data-parallel training behaviors strictly, traces the consistency-relevant factors carefully, utilizes the deep learning characteristics for EasyScaleThread abstraction and fast context-switching. To utilize heterogeneous cluster, EasyScale dynamically assigns workers based on the intra-/inter-job schedulers, minimizing load imbalance and maximizing aggregated job throughput. Deployed in an online serving cluster, EasyScale powers the training jobs to utilize idle GPUs opportunistically, improving overall cluster utilization by 62.1%.

Motivation & Objective

• To address the critical challenge of inconsistent model accuracy in elastic deep learning training due to dynamic resource allocation and GPU heterogeneity.
• To decouple the model training process from resource allocation, preserving the developer’s original hyperparameters and training procedure.
• To achieve bitwise determinism and reproducibility in distributed training under elasticity, even across different GPU types.
• To maximize cluster utilization by opportunistically utilizing idle GPUs without sacrificing model accuracy.
• To enable seamless integration of elastic training into production GPU clusters without requiring changes to model design or training workflows.

Proposed method

• Introduces the EasyScaleThread (EST) abstraction to encapsulate training state and ensure consistent execution across dynamic GPU allocations.
• Traces and controls consistency-relevant factors across the deep learning software stack, including cuBLAS, NCCL, and CUDA kernels, to eliminate non-determinism.
• Employs a two-level scheduling strategy: intra-job scheduler for dynamic worker reassignment within a job, and inter-job scheduler for load balancing across heterogeneous GPU clusters.
• Leverages deep learning characteristics such as gradient accumulation and mini-batch processing to enable fast context switching and low overhead during scale-in/scale-out.
• Uses a deterministic execution model that ensures bitwise-identical results across runs, even when the number of GPUs or GPU types change.
• Integrates with existing DDP-based training frameworks (e.g., PyTorch DDP) without requiring changes to model architecture or hyperparameters.

Experimental results

Research questions

• RQ1 Can elastic deep learning training maintain consistent model accuracy across varying numbers of GPUs and heterogeneous GPU types?
• RQ2 How can model determinism and reproducibility be preserved during dynamic scale-in/scale-out operations in distributed training?
• RQ3 What system-level abstractions and scheduling mechanisms are required to decouple training behavior from resource allocation while maintaining performance?
• RQ4 To what extent can idle GPU resources be utilized in production clusters without degrading model accuracy?
• RQ5 Can a deterministic elastic training system be deployed at scale in real-world online serving environments?

Key findings

• EasyScale improves overall cluster GPU utilization by 62.1% by opportunistically using idle GPUs in a production online serving environment.
• The system successfully scales in and out within 5 minutes after inference jobs release GPUs, demonstrating rapid responsiveness to resource availability.
• No EasyScale training job failed during a one-day production run, despite 362 preemptions, confirming high reliability and fault tolerance.
• The system maintains consistent model accuracy across different numbers of GPUs and GPU types, including A100 and older generations, by eliminating non-deterministic behaviors in the software stack.
• On average, 459 idle GPUs were utilized per day by EasyScale jobs, which were quickly reclaimed when needed for high-priority inference workloads.
• The use of EST abstraction and deterministic execution ensures bitwise-identical training results, even under dynamic resource changes, enabling full reproducibility.