[Paper Review] Investigating Warp Size Impact in GPUs

Ahmad Lashgar, Amirali Baniasadi+1 more|arXiv (Cornell University)|May 22, 2012

Parallel Computing and Optimization Techniques14 references3 citations

TL;DR

This paper investigates the impact of warp size on GPU performance by comparing two GPU designs: one with small warps and enhanced memory coalescing (SW+), and another with large warps and an MIMD-like control-flow mechanism to mitigate branch divergence (LW+). The authors find that coalescing-enhanced small warp GPUs outperform control-flow-optimized large warp GPUs, making small warps with improved memory access a more effective design strategy.

ABSTRACT

There are a number of design decisions that impact a GPU's performance. Among such decisions deciding the right warp size can deeply influence the rest of the design. Small warps reduce the performance penalty associated with branch divergence at the expense of a reduction in memory coalescing. Large warps enhance memory coalescing significantly but also increase branch divergence. This leaves designers with two choices: use a small warps and invest in finding new solutions to enhance coalescing or use large warps and address branch divergence employing effective control-flow solutions. In this work our goal is to investigate the answer to this question. We analyze warp size impact on memory coalescing and branch divergence. We use our findings to study two machines: a GPU using small warps but equipped with excellent memory coalescing (SW+) and a GPU using large warps but employing an MIMD engine immune from control-flow costs (LW+). Our evaluations show that building coalescing-enhanced small warp GPUs is a better approach compared to pursuing a control-flow enhanced large warp GPU.

Motivation & Objective

• To analyze the trade-offs between warp size, memory coalescing, and branch divergence in GPU architectures.
• To evaluate the performance implications of using small warps with enhanced memory coalescing versus large warps with advanced control-flow handling.
• To determine whether investing in memory coalescing improvements for small warps is more beneficial than optimizing control-flow for large warps.
• To compare two hypothetical GPU designs: SW+ (small warp, coalescing-optimized) and LW+ (large warp, MIMD-based control-flow)

Proposed method

• The authors analyze the effects of warp size on memory coalescing and branch divergence using analytical modeling and simulation.
• They design and evaluate two GPU architectures: SW+ (small warps with enhanced memory coalescing) and LW+ (large warps with MIMD-style execution to reduce control-flow penalties).
• The evaluation uses a technical report and simulation-based performance comparison across representative workloads.
• The study focuses on quantifying the performance impact of branch divergence and memory coalescing efficiency under different warp sizes.
• The authors use a combination of theoretical analysis and performance modeling to compare the two GPU designs.

Experimental results

Research questions

• RQ1 How does warp size affect memory coalescing efficiency in GPU architectures?
• RQ2 What is the performance impact of branch divergence in large versus small warps?
• RQ3 Which design approach yields better overall GPU performance: enhancing memory coalescing in small warps or improving control-flow handling in large warps?
• RQ4 Can a small warp GPU with superior memory coalescing outperform a large warp GPU with advanced control-flow mechanisms?

Key findings

• The SW+ design, featuring small warps with enhanced memory coalescing, outperforms the LW+ design with large warps and MIMD-based control-flow handling.
• Small warps reduce branch divergence penalties, which significantly improves performance when combined with effective memory coalescing.
• The performance gain from improved memory coalescing in small warps outweighs the benefits of reduced control-flow overhead in large warps.
• The study concludes that investing in memory coalescing optimization for small warps is a more effective design strategy than focusing on control-flow enhancements for large warps.