[Paper Review] Towards Latency-aware DNN Optimization with GPU Runtime Analysis and Tail Effect Elimination

Fuxun Yu, Zirui Xu+13 more|arXiv (Cornell University)|Nov 8, 2020

Advanced Neural Network Applications19 references4 citations

TL;DR

This paper proposes a GPU runtime-aware DNN optimization methodology that identifies and eliminates the GPU tail effect—caused by inefficient thread block scheduling and underutilization—through full-stack analysis of GPU execution patterns. By adapting DNN model configurations to GPU wave-level execution, it achieves 11%–27% latency reduction and 2.5%–4.0% accuracy improvement over SOTA pruning and NAS methods on both high-end and embedded GPUs.

ABSTRACT

Despite the superb performance of State-Of-The-Art (SOTA) DNNs, the increasing computational cost makes them very challenging to meet real-time latency and accuracy requirements. Although DNN runtime latency is dictated by model property (e.g., architecture, operations), hardware property (e.g., utilization, throughput), and more importantly, the effective mapping between these two, many existing approaches focus only on optimizing model property such as FLOPS reduction and overlook the mismatch between DNN model and hardware properties. In this work, we show that the mismatch between the varied DNN computation workloads and GPU capacity can cause the idle GPU tail effect, leading to GPU under-utilization and low throughput. As a result, the FLOPs reduction cannot bring effective latency reduction, which causes sub-optimal accuracy versus latency trade-offs. Motivated by this, we propose a GPU runtime-aware DNN optimization methodology to eliminate such GPU tail effect adaptively on GPU platforms. Our methodology can be applied on top of existing SOTA DNN optimization approaches to achieve better latency and accuracy trade-offs. Experiments show 11%-27% latency reduction and 2.5%-4.0% accuracy improvement over several SOTA DNN pruning and NAS methods, respectively

Motivation & Objective

• To address the gap between theoretical FLOPs reduction and actual runtime latency gains in DNNs due to hardware-software mismatch.
• To identify the root cause of inconsistent latency reduction despite FLOPs reduction, specifically the GPU tail effect from suboptimal thread scheduling.
• To develop a methodology that adapts DNN model structures to GPU runtime characteristics for improved latency and accuracy trade-offs.
• To generalize the approach across diverse GPU platforms, including high-end and embedded systems.

Proposed method

• Conducts full-stack GPU runtime analysis by examining individual GPU cores and threads to uncover the intrinsic causes of latency staircasing.
• Re-defines the 'GPU tail effect' as a result of inefficient wave-level GPU execution due to mismatched thread block sizes and kernel launch patterns.
• Proposes thread-adaptive DNN deployment that reconfigures model structures (e.g., filter counts, batch size, filter shapes) to align with GPU wave boundaries and improve utilization.
• Introduces a GPU efficiency guideline that maps DNN layer configurations to optimal GPU execution cycles, minimizing idle cycles and throughput fluctuations.
• Applies the methodology on top of existing SOTA DNN optimization techniques (e.g., pruning, NAS) without requiring GPU-specific code changes.
• Validates the approach across multiple DNNs (VGG16, ResNet56) and GPU platforms (P6000, Jetson Nano), demonstrating generalizability.

Experimental results

Research questions

• RQ1 Why does FLOPs reduction not consistently translate to latency reduction in DNN inference on GPUs?
• RQ2 What causes the non-linear 'latency staircase' pattern observed during DNN model pruning and reconfiguration?
• RQ3 How does GPU hardware utilization and throughput fluctuate during DNN inference, and what role does this play in performance degradation?
• RQ4 Can the GPU tail effect be systematically identified and mitigated through model-level configuration adjustments?
• RQ5 To what extent can the proposed optimization methodology generalize across different GPU platforms and DNN architectures?

Key findings

• The latency staircase pattern—indicating inconsistent latency reduction despite FLOPs reduction—persists across varying batch sizes, input resolutions, filter shapes, and number of filters.
• The GPU tail effect arises from inefficient thread block scheduling and wave-level GPU execution, causing idle cycles and underutilization even when FLOPs are reduced.
• On the P6000 GPU, the method achieves 27.2% latency reduction for VGG16 and 9.0% for ResNet56 while maintaining or slightly improving accuracy.
• On the Jetson Nano embedded GPU, the method achieves 20.5% latency reduction for VGG16 and 13.3%–17.1% for ResNet56, demonstrating strong cross-platform generalizability.
• The approach improves accuracy-latency trade-offs by 2.5%–4.0% over SOTA pruning and NAS methods, even when applied as a post-optimization layer.
• The latency staircase is most pronounced in dense convolutional layers with large filter counts and high batch sizes, but is mitigated in lightweight depthwise convolutions.