[Paper Review] Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads

Dharma D. Shukla, Muthian Sivathanu+24 more|arXiv (Cornell University)|Feb 16, 2022

Stochastic Gradient Optimization Techniques4 citations

TL;DR

Singularity is a planet-scale, transparent scheduler for AI workloads that enables fully preemptive and elastic execution of unmodified deep learning jobs across global GPU and accelerator fleets. By decoupling jobs from physical devices via device proxies and replica splicing, it achieves low-overhead, on-demand checkpointing and live migration with negligible performance impact, enabling high utilization and strict SLAs without user code changes.

ABSTRACT

Lowering costs by driving high utilization across deep learning workloads is a crucial lever for cloud providers. We present Singularity, Microsoft's globally distributed scheduling service for highly-efficient and reliable execution of deep learning training and inference workloads. At the heart of Singularity is a novel, workload-aware scheduler that can transparently preempt and elastically scale deep learning workloads to drive high utilization without impacting their correctness or performance, across a global fleet of AI accelerators (e.g., GPUs, FPGAs). All jobs in Singularity are preemptable, migratable, and dynamically resizable (elastic) by default: a live job can be dynamically and transparently (a) preempted and migrated to a different set of nodes, cluster, data center or a region and resumed exactly from the point where the execution was preempted, and (b) resized (i.e., elastically scaled-up/down) on a varying set of accelerators of a given type. Our mechanisms are transparent in that they do not require the user to make any changes to their code or require using any custom libraries that may limit flexibility. Additionally, our approach significantly improves the reliability of deep learning workloads. We show that the resulting efficiency and reliability gains with Singularity are achieved with negligible impact on the steady-state performance. Finally, our design approach is agnostic of DNN architectures and handles a variety of parallelism strategies (e.g., data/pipeline/model parallelism).

Motivation & Objective

• To eliminate resource idling and maximize throughput across a global fleet of AI accelerators (GPUs, FPGAs) by enabling opportunistic use of spare capacity.
• To provide strong, job-level SLAs for training and inference workloads despite dynamic resource allocation and preemption.
• To improve reliability of long-running DNN jobs by enabling resumption from preemption points, avoiding costly restarts.
• To make elasticity and preemption first-class, transparent features for all unmodified DNN jobs, without requiring custom libraries or hyperparameter tuning.
• To decouple job execution from physical hardware using device proxies and replica splicing, enabling dynamic migration and resizing with minimal overhead.

Proposed method

• Introduces a device-proxy abstraction that isolates GPU device state from worker processes, enabling clean checkpointing and migration.
• Employs replica splicing to time-slice multiple DNN workers on the same physical device with full memory access and low overhead, exploiting memory similarity across workers.
• Uses a synchronization barrier at mini-batch boundaries to create consistent, application-level checkpoints for live migration.
• Enables dynamic resizing of jobs by remapping the same number of logical workers to varying numbers of physical accelerators without changing the job’s configuration.
• Leverages CRIU for efficient, transparent process checkpointing and migration, with semantics-aware optimizations to reduce overhead.
• Treats preemption and elasticity as first-class scheduling primitives, integrated into a hierarchical, globally-distributed scheduler.

Experimental results

Research questions

• RQ1 How can deep learning workloads be made fully preemptible and migratable across clusters and regions without requiring application-level changes?
• RQ2 Can transparent elasticity be achieved for unmodified DNN training jobs without altering the training loop or requiring custom libraries?
• RQ3 How can low-overhead, consistent checkpointing be achieved in a heterogeneous, distributed DNN environment with minimal performance impact?
• RQ4 What architectural mechanisms enable high utilization and strict SLAs in a globally-distributed AI infrastructure without resource fragmentation?
• RQ5 Can a system achieve both high statistical efficiency and high throughput in DNN training by dynamically scaling resources transparently?

Key findings

• Singularity enables transparent preemption and migration of unmodified DNN jobs across regions and data centers with negligible steady-state performance overhead.
• The use of device proxies and replica splicing allows multiple workers to share a single GPU with full memory access and minimal performance degradation.
• All jobs are automatically checkpointed at safe synchronization points (e.g., end of mini-batch), enabling reliable resumption after preemption.
• Elasticity is fully transparent: jobs maintain the same number of workers and hyperparameters, while being dynamically remapped to different numbers of physical accelerators.
• The system achieves high utilization across a global fleet of hundreds of thousands of GPUs by opportunistically using spare capacity without violating SLAs.
• By decoupling jobs from hardware and enabling work-conserving migration, Singularity reduces wasted compute and improves reliability for long-running training jobs.