[Paper Review] Vis-TOP: Visual Transformer Overlay Processor

Wei Hu, Dian Xu+3 more|arXiv (Cornell University)|Oct 21, 2021

Advanced Image and Video Retrieval Techniques28 references4 citations

TL;DR

Vis-TOP is a reconfigurable overlay processor for visual Transformer models that uses a three-layer, two-level transformation architecture to enable flexible, efficient hardware mapping across diverse vision Transformer models. It achieves 1.5x higher throughput than GPU and 2.2x–11.7x higher throughput per DSP than existing accelerators, offering a power- and cost-effective solution for edge AI inference.

ABSTRACT

In recent years, Transformer has achieved good results in Natural Language Processing (NLP) and has also started to expand into Computer Vision (CV). Excellent models such as the Vision Transformer and Swin Transformer have emerged. At the same time, the platform for Transformer models was extended to embedded devices to meet some resource-sensitive application scenarios. However, due to the large number of parameters, the complex computational flow and the many different structural variants of Transformer models, there are a number of issues that need to be addressed in its hardware design. This is both an opportunity and a challenge. We propose Vis-TOP (Visual Transformer Overlay Processor), an overlay processor for various visual Transformer models. It differs from coarse-grained overlay processors such as CPU, GPU, NPE, and from fine-grained customized designs for a specific model. Vis-TOP summarizes the characteristics of all visual Transformer models and implements a three-layer and two-level transformation structure that allows the model to be switched or changed freely without changing the hardware architecture. At the same time, the corresponding instruction bundle and hardware architecture are designed in three-layer and two-level transformation structure. After quantization of Swin Transformer tiny model using 8-bit fixed points (fix_8), we implemented an overlay processor on the ZCU102. Compared to GPU, the TOP throughput is 1.5x higher. Compared to the existing Transformer accelerators, our throughput per DSP is between 2.2x and 11.7x higher than others. In a word, the approach in this paper meets the requirements of real-time AI in terms of both resource consumption and inference speed. Vis-TOP provides a cost-effective and power-effective solution based on reconfigurable devices for computer vision at the edge.

Motivation & Objective

• To address the challenge of deploying diverse, resource-intensive visual Transformer models on edge devices with limited power and area.
• To overcome the limitations of general-purpose accelerators (e.g., GPU) and fixed-architecture accelerators by enabling reconfigurability without hardware redesign.
• To achieve high throughput and efficient resource utilization through a co-designed software-hardware stack for vision Transformers.
• To support model evolution and new architectures via programmable instruction bundles and modular hardware components.

Proposed method

• The paper proposes a three-layer, two-level transformation architecture: model layer, container layer, and component layer, enabling transparent mapping from neural network operations to hardware.
• It introduces a custom instruction bundle that supports dynamic reconfiguration of computation and data flow across different Transformer blocks.
• The hardware is designed with universal modular units (fixed batch size) and variable modular units (reconfigurable structure) to support diverse attention and feed-forward operations.
• The system performs model slicing from a hardware perspective, enabling fine-grained parallelism and pipelining across attention and FFN layers.
• The design supports 8-bit fixed-point quantization (fix_8) of the Swin Transformer tiny model for efficient deployment on ZCU102 FPGA.
• Throughput and DSP efficiency are optimized via instruction-level scheduling and hardware resource reuse across model variants.

Experimental results

Research questions

• RQ1 How can a single hardware architecture efficiently support multiple visual Transformer models with varying structures?
• RQ2 What is the optimal balance between hardware reconfigurability and computational efficiency for vision Transformer inference on FPGAs?
• RQ3 Can an overlay processor achieve higher throughput per DSP than existing specialized accelerators for Transformers?
• RQ4 How does the proposed three-layer, two-level transformation model enable transparent mapping from software to hardware?
• RQ5 What is the impact of modular, composable hardware units on resource utilization and inference speed?

Key findings

• Vis-TOP achieves 11.83 inference frames per second on the ZCU102 FPGA for the quantized Swin Transformer tiny model.
• Compared to GPU (Tesla T4), Vis-TOP delivers 1.5x higher throughput, with 55.86 inference/sec vs. 55.86 inference/sec.
• Vis-TOP achieves 2.2x to 11.7x higher throughput per DSP slice than other accelerators, including FTRANS, NPE, and FQ-BERT.
• The throughput per DSP is 0.1515 for Vis-TOP, significantly outperforming FTRANS (0.0148), NPE (0.0669), and FQ-BERT (0.0129).
• Vis-TOP uses only 558 DSP slices, far fewer than FTRANS (6840), demonstrating high resource efficiency despite high throughput.
• The design enables model reconfiguration without hardware changes, supporting future vision Transformer models via instruction updates or new module interfaces.