[Paper Review] Full Stack Optimization of Transformer Inference: a Survey

Sehoon Kim, Coleman Hooper+10 more|arXiv (Cornell University)|Feb 27, 2023

Ferroelectric and Negative Capacitance DevicesEngineering27 citations

TL;DR

This survey analyzes full-stack approaches to efficient Transformer inference and demonstrates a Gemmini case study with up to 88.7× speedup and minimal degradation.

ABSTRACT

Recent advances in state-of-the-art DNN architecture design have been moving toward Transformer models. These models achieve superior accuracy across a wide range of applications. This trend has been consistent over the past several years since Transformer models were originally introduced. However, the amount of compute and bandwidth required for inference of recent Transformer models is growing at a significant rate, and this has made their deployment in latency-sensitive applications challenging. As such, there has been an increased focus on making Transformer models more efficient, with methods that range from changing the architecture design, all the way to developing dedicated domain-specific accelerators. In this work, we survey different approaches for efficient Transformer inference, including: (i) analysis and profiling of the bottlenecks in existing Transformer architectures and their similarities and differences with previous convolutional models; (ii) implications of Transformer architecture on hardware, including the impact of non-linear operations such as Layer Normalization, Softmax, and GELU, as well as linear operations, on hardware design; (iii) approaches for optimizing a fixed Transformer architecture; (iv) challenges in finding the right mapping and scheduling of operations for Transformer models; and (v) approaches for optimizing Transformer models by adapting the architecture using neural architecture search. Finally, we perform a case study by applying the surveyed optimizations on Gemmini, the open-source, full-stack DNN accelerator generator, and we show how each of these approaches can yield improvements, compared to previous benchmark results on Gemmini. Among other things, we find that a full-stack co-design approach with the aforementioned methods can result in up to 88.7x speedup with a minimal performance degradation for Transformer inference.

Motivation & Objective

• Analyze runtime bottlenecks and workload characteristics of Transformer architectures.
• Examine hardware impact of nonlinear and linear Transformer operations on inference efficiency.
• Survey optimization techniques for fixed Transformer architectures (e.g., pruning, quantization).
• Explore scheduling/mapping challenges for Transformer workloads across hardware.
• Investigate neural architecture search to tailor Transformers for hardware efficiency.

Proposed method

• Survey Transformer runtime characteristics and profiling of bottlenecks (Sec. 2).
• Analyze hardware implications of non-linear ops (LayerNorm, Softmax, GELU) and linear ops (matmuls) for accelerators (Sec. 3).
• Review optimization techniques for fixed architectures (pruning, quantization) (Sec. 4).
• Discuss operation mapping and scheduling challenges (Sec. 5).
• Describe neural architecture search approaches to adapt Transformer architectures for hardware efficiency (Sec. 6).
• Present a case study applying surveyed optimizations on Gemmini and report performance implications (Sec. 3.4, Fig. 14, Sec. 5.5).

Experimental results

Research questions

• RQ1 What are the runtime bottlenecks of Transformer encoders and decoders on hardware?
• RQ2 How do non-linear operations in Transformers affect accelerator design and utilization?
• RQ3 What optimization strategies maximize performance for a fixed Transformer architecture?
• RQ4 What scheduling/mapping decisions most influence Transformer inference latency?
• RQ5 Can neural architecture search yield hardware-efficient Transformer variants, and what are the trade-offs?

Key findings

• A full-stack co-design approach can yield up to 88.7× speedup with minimal performance degradation for Transformer inference on Gemmini.
• Gemmini’s CNN-optimized architecture is not well-suited for Transformer inference due to time spent in floating-point non-linear and quantization/dequantization operations, potentially causing under 1% hardware utilization if not addressed.
• For Transformer accelerators, larger accumulator size with smaller scratchpad size often improves performance compared to CNN-optimized designs (≈36% latency improvement in the reported case).
• Scheduling matmuls in Transformers is as challenging as in CNNs, with best/worst solutions differing by up to four orders of magnitude (Sec. 5.5.1).
• Fusing LayerNorm with preceding matmul introduces tile-size constraints that can negate fusion gains in some scenarios (Sec. 5.5.2).