[Paper Review] An Efficient FPGA-Based Accelerator for Swin Transformer

Zhiyang Liu, Pengyu Yin+1 more|arXiv (Cornell University)|Aug 26, 2023

Advanced Memory and Neural ComputingEngineering3 citations

TL;DR

This paper presents an efficient FPGA-based accelerator for Swin Transformer that replaces Layer Normalization with Batch Normalization to enable hardware fusion, designs approximate hardware-friendly units for Softmax and GELU, and employs a unified Matrix Multiplication Unit for all linear operations. The accelerator achieves up to 1.76x speedup and 20.45x higher energy efficiency than a CPU, outperforming existing FPGA accelerators in speed and efficiency.

ABSTRACT

Since introduced, Swin Transformer has achieved remarkable results in the field of computer vision, it has sparked the need for dedicated hardware accelerators, specifically catering to edge computing demands. For the advantages of flexibility, low power consumption, FPGAs have been widely employed to accelerate the inference of convolutional neural networks (CNNs) and show potential in Transformer-based models. Unlike CNNs, which mainly involve multiply and accumulate (MAC) operations, Transformer involve non-linear computations such as Layer Normalization (LN), Softmax, and GELU. These nonlinear computations do pose challenges for accelerator design. In this paper, to propose an efficient FPGA-based hardware accelerator for Swin Transformer, we focused on using different strategies to deal with these nonlinear calculations and efficiently handling MAC computations to achieve the best acceleration results. We replaced LN with BN, Given that Batch Normalization (BN) can be fused with linear layers during inference to optimize inference efficiency. The modified Swin-T, Swin-S, and Swin-B respectively achieved Top-1 accuracy rates of 80.7%, 82.7%, and 82.8% in ImageNet. Furthermore, We employed strategies for approximate computation to design hardware-friendly architectures for Softmax and GELU computations. We also designed an efficient Matrix Multiplication Unit to handle all linear computations in Swin Transformer. As a conclude, compared with CPU (AMD Ryzen 5700X), our accelerator achieved 1.76x, 1.66x, and 1.25x speedup and achieved 20.45x, 18.60x, and 14.63x energy efficiency (FPS/power consumption) improvement on Swin-T, Swin-S, and Swin-B models, respectively. Compared to GPU (Nvidia RTX 2080 Ti), we achieved 5.05x, 4.42x, and 3.00x energy efficiency improvement respectively. As far as we know, the accelerator we proposed is the fastest FPGA-based accelerator for Swin Transformer.

Motivation & Objective

• To address the challenge of accelerating Swin Transformer on FPGAs, which involves complex nonlinear operations like Softmax and GELU that are resource-intensive and latency-heavy.
• To overcome the inefficiency of Layer Normalization (LN) in FPGA hardware by replacing it with Batch Normalization (BN), enabling fusion with linear layers for improved inference efficiency.
• To design hardware-friendly approximations for nonlinear functions (Softmax and GELU) using shift, add, and multiply operations to reduce FPGA resource usage and latency.
• To develop a unified Matrix Multiplication Unit (MMU) that efficiently handles all linear computations in Swin Transformer, leveraging the structural characteristics of its matrix operations.
• To achieve high-speed, low-power inference on edge devices by optimizing the entire accelerator stack for FPGA deployment.

Proposed method

• Replaced Layer Normalization (LN) with Batch Normalization (BN) in Swin-T, Swin-S, and Swin-B models to allow fusion with linear layers, reducing computational overhead and enabling efficient hardware mapping.
• Transformed the PatchEmbed convolution into matrix multiplication to unify linear computation handling and improve hardware utilization.
• Designed approximate hardware architectures for Softmax and GELU using base-2 exponentiation and shift-add operations, minimizing reliance on complex divisions and transcendental functions.
• Implemented a single, efficient Matrix Multiplication Unit (MMU) that supports all matrix multiplications in Swin Transformer, optimized for the specific matrix shapes and data types used.
• Used fixed-point arithmetic (Fix16) to reduce resource consumption while maintaining high accuracy, with model accuracy loss of less than 0.5% on ImageNet.
• Conducted power and performance measurements using HWiNFO64 (CPU), MSI Afterburner (GPU), and Vivado Power Report (FPGA) to evaluate energy efficiency.

Experimental results

Research questions

• RQ1 Can replacing Layer Normalization with Batch Normalization in Swin Transformer enable efficient hardware fusion and reduce FPGA resource usage without significant accuracy loss?
• RQ2 How can nonlinear operations like Softmax and GELU be approximated using only shift, add, and multiply operations to reduce FPGA resource consumption and latency?
• RQ3 What is the optimal design for a unified Matrix Multiplication Unit that supports all linear operations in Swin Transformer with minimal area and high throughput?
• RQ4 How does the proposed FPGA accelerator compare in speed and energy efficiency to CPU and GPU baselines for Swin Transformer inference?
• RQ5 Can the proposed accelerator achieve better performance than existing FPGA-based Swin Transformer accelerators in terms of throughput, power efficiency, and resource utilization?

Key findings

• The modified Swin-T, Swin-S, and Swin-B models achieved Top-1 accuracy of 80.7%, 82.7%, and 82.8% on ImageNet, respectively, with only a minor drop compared to the original models.
• The FPGA accelerator achieved 1.76x, 1.66x, and 1.25x speedup over the AMD Ryzen 5700X CPU for Swin-T, Swin-S, and Swin-B, respectively.
• The accelerator achieved 20.45x, 18.60x, and 14.63x higher energy efficiency (FPS/watt) than the CPU for the three models.
• Compared to the Nvidia RTX 2080 Ti GPU, the accelerator achieved 5.05x, 4.42x, and 3.00x higher energy efficiency, respectively.
• The accelerator used only 10.69W, 10.69W, and 11.11W power for Swin-T, Swin-S, and Swin-B, respectively, significantly lower than the GPU’s 240W.
• The proposed accelerator outperformed existing FPGA accelerators in throughput and energy efficiency, achieving 431.2 GOPS for Swin-T with 1727 DSPs, and being the fastest reported FPGA-based Swin Transformer accelerator to date.