[Paper Review] MuxFlow: Efficient and Safe GPU Sharing in Large-Scale Production Deep Learning Clusters

Yihao Zhao, Xin Liu+6 more|arXiv (Cornell University)|Mar 24, 2023

Brain Tumor Detection and ClassificationNeuroscience3 citations

TL;DR

MuxFlow is a production-ready GPU cluster system that enables efficient and safe space-sharing of GPUs for mixed online and offline deep learning workloads. By combining a two-level performance protection mechanism, a mixed error-handling strategy, and dynamic SM allocation with matching-based scheduling, MuxFlow increases GPU utilization from 26% to 76% in real-world deployment at CompanyX’s 20,000+ GPU cluster.

ABSTRACT

Large-scale GPU clusters are widely-used to speed up both latency-critical (online) and best-effort (offline) deep learning (DL) workloads. However, most DL clusters either dedicate each GPU to one workload or share workloads in time, leading to very low GPU resource utilization. We present MuxFlow, the first production cluster system that supports efficient and safe space-sharing for DL workloads. NVIDIA MPS provides an opportunity to share multiple workloads in space on widely-deployed NVIDIA GPUs, but it cannot guarantee the performance and safety of online workloads. MuxFlow introduces a two-level protection mechanism for memory and computation to guarantee the performance of online workloads. Based on our practical error analysis, we design a mixed error-handling mechanism to guarantee the safety of online workloads. MuxFlow further proposes dynamic streaming multiprocessor (SM) allocation and matching-based scheduling to improve the efficiency of offline workloads. MuxFlow has been deployed at CompanyX's clusters with more than 20,000 GPUs. The deployment results indicate that MuxFlow substantially improves the GPU utilization from 26$\%$ to 76$\%$, SM activity from 16$\%$ to 33$\%$, and GPU memory from 42$\%$ to 48$\%$.

Motivation & Objective

• To address the low GPU utilization in large-scale deep learning clusters where GPUs are either dedicated to single workloads or time-shared, leading to underutilization.
• To enable space-sharing of GPUs via NVIDIA MPS while guaranteeing performance isolation for latency-sensitive online workloads.
• To ensure safety of online workloads by mitigating error propagation, especially from container signals like SIGINT and SIGTERM in Kubernetes environments.
• To improve efficiency of offline workloads through dynamic streaming multiprocessor (SM) allocation and intelligent workload pairing via bipartite matching.
• To deploy a practical, production-grade system that balances performance, safety, and utilization in real-world clusters.

Proposed method

• Introduces xCUDA, a runtime mechanism that enforces per-workload limits on GPU memory and kernel launch frequency to constrain offline workload resource consumption.
• Employs a GPU-level SysMonitor that uses multi-dimensional metrics to detect performance degradation risks and triggers eviction of offline workloads when necessary.
• Designs a mixed error-handling mechanism that intercepts and gracefully handles SIGINT/SIGTERM signals (responsible for 99% of propagated errors) and resets CUDA contexts for corner-case errors.
• Proposes dynamic SM allocation where the SM percentage used by offline workloads is adjusted complementarily to online workloads, minimizing performance impact.
• Models offline workload pairing as a maximum weighted bipartite matching problem, using a deep learning-based predictor to estimate interference and the KM algorithm to find optimal pairings.
• Integrates all components into a full-stack cluster system deployed in production at CompanyX, supporting thousands of GPUs with minimal changes to existing infrastructure.

Experimental results

Research questions

• RQ1 How can GPU space-sharing be made safe and efficient in large-scale production deep learning clusters without degrading the performance of low-latency online workloads?
• RQ2 What mechanisms can effectively prevent error propagation from offline to online workloads in a shared GPU environment?
• RQ3 How can dynamic SM allocation improve the utilization of GPU resources while maintaining acceptable performance for online workloads?
• RQ4 What is the optimal strategy for pairing offline workloads with online workloads to maximize overall efficiency in a shared GPU cluster?
• RQ5 Can a practical, production-ready system be built that combines space-sharing, performance isolation, and safety guarantees using existing GPU virtualization primitives like NVIDIA MPS?

Key findings

• MuxFlow increased GPU utilization from 26% to 76% in production clusters with over 20,000 GPUs, significantly improving resource efficiency.
• SM activity rose from 16% to 33%, indicating better utilization of GPU compute units under shared workloads.
• GPU memory utilization improved from 42% to 48%, demonstrating effective memory management across mixed workloads.
• The mixed error-handling mechanism successfully mitigated 99% of propagated errors, primarily caused by SIGINT/SIGTERM signals during container shutdown.
• The matching-based scheduling with DL-based interference prediction achieved high efficiency in workload pairing, minimizing performance degradation for online workloads.
• Dynamic SM allocation enabled complementary resource usage between online and offline workloads, allowing offline tasks to utilize idle GPU capacity with minimal impact on online latency.