[Paper Review] VirtualFlow: Decoupling Deep Learning Models from the Underlying Hardware

Andrew Or, Haoyu Zhang+1 more|arXiv (Cornell University)|Sep 20, 2020

Advanced Neural Network Applications51 references4 citations

TL;DR

VirtualFlow decouples deep learning models from hardware by introducing virtual node processing, which splits mini-batches across virtual nodes mapped to physical accelerators, enabling consistent training across diverse hardware, resource elasticity, and heterogeneous training without hyperparameter tuning. It achieves up to 48% faster job completion and 42% higher throughput with minimal changes to existing frameworks.

ABSTRACT

State-of-the-art deep learning systems such as TensorFlow and PyTorch tightly couple the model with the underlying hardware. This coupling requires the user to modify application logic in order to run the same job across a different set of resources, thereby limiting the choice of hardware for a given workload and potentially forcing the user to forgo more efficient hardware configurations. We propose VirtualFlow, a system leveraging a novel abstraction called virtual node processing to decouple the model from the hardware. In each step of training or inference, the batch of input data is split across virtual nodes instead of hardware accelerators (e.g. GPUs and TPUs). Mapping multiple virtual nodes to each accelerator and processing them sequentially effectively time slices the batch, thereby allowing users to reduce the memory requirement of their workloads and mimic large batch sizes on small clusters. Using this technique, VirtualFlow enables many new use cases, such as reproducing training results across different hardware, resource elasticity, and heterogeneous training. In our evaluation, our implementation of VirtualFlow for TensorFlow achieved strong convergence guarantees across different hardware with out-of-the-box hyperparameters, up to 48% lower job completion times with resource elasticity, and up to 42% higher throughput with heterogeneous training.

Motivation & Objective

• Address the tight coupling between deep learning models and hardware in frameworks like TensorFlow and PyTorch, which forces users to retune hyperparameters across different hardware.
• Enable consistent model convergence across heterogeneous and dynamic hardware configurations without modifying model specifications or hyperparameters.
• Support resource elasticity and heterogeneous training by allowing batch processing to be time-sliced across accelerators, reducing memory pressure and enabling efficient use of mixed GPU types.
• Overcome the limitations of existing systems that require job restarts for resource adjustments and cannot handle mixed accelerator types.
• Facilitate reproducibility and experimentation by allowing users to train models on small clusters with the same hyperparameters used on large clusters.

Proposed method

• Introduce virtual nodes as a new abstraction to decouple model execution from physical hardware, where each batch is split into equally sized virtual nodes.
• Map multiple virtual nodes to the same physical accelerator and execute them sequentially, effectively time-slicing the batch to reduce memory usage and simulate larger batch sizes.
• Integrate virtual node processing into TensorFlow with minimal changes, preserving model graph structure and hyperparameters across different hardware setups.
• Enable resource elasticity by dynamically resizing the number of virtual nodes per accelerator during training without interrupting the job.
• Support heterogeneous training by mapping virtual nodes to different accelerator types (e.g., V100, TPU) within the same job, leveraging underutilized hardware.
• Combine virtual nodes with model parallelism to reduce resource requirements in large-scale model training, unrolling data parallel pipelines into sequential passes.

Experimental results

Research questions

• RQ1 Can deep learning models be trained with consistent convergence across diverse hardware configurations without hyperparameter retuning?
• RQ2 Can resource elasticity be achieved in deep learning training without interrupting jobs or altering batch size?
• RQ3 Can heterogeneous GPU clusters be effectively utilized for a single training job without requiring homogeneous hardware?
• RQ4 Can virtual node processing reduce memory pressure and enable large-batch training on small clusters?
• RQ5 Can virtual nodes be integrated into existing deep learning frameworks with minimal modifications while preserving training stability?

Key findings

• VirtualFlow achieved strong convergence guarantees across different hardware configurations using out-of-the-box hyperparameters, enabling consistent training results regardless of underlying hardware.
• With resource elasticity, VirtualFlow reduced job completion times by up to 48% by dynamically adjusting virtual node allocation without interrupting training.
• In heterogeneous training scenarios, VirtualFlow increased throughput by up to 42% by efficiently utilizing mixed GPU types within the same job.
• The system enabled reproducible training across clusters of varying sizes and hardware types, eliminating the need for hyperparameter retuning.
• VirtualFlow reduced memory pressure by time-slicing batches across virtual nodes, allowing large-batch training on small clusters.
• The integration of virtual nodes with model parallelism reduced resource requirements by half in evaluated workloads, enabling more efficient scaling of large models.