[Paper Review] Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Ze Liu, Yutong Lin+6 more|arXiv (Cornell University)|Mar 25, 2021

Advanced Neural Network Applications368 citations

TL;DR

Swin Transformer introduces a hierarchical vision Transformer with shifted windows, achieving linear complexity and strong results across image classification, object detection, and semantic segmentation.

ABSTRACT

This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, we propose a hierarchical Transformer whose representation is computed with extbf{S}hifted extbf{win}dows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. These qualities of Swin Transformer make it compatible with a broad range of vision tasks, including image classification (87.3 top-1 accuracy on ImageNet-1K) and dense prediction tasks such as object detection (58.7 box AP and 51.1 mask AP on COCO test-dev) and semantic segmentation (53.5 mIoU on ADE20K val). Its performance surpasses the previous state-of-the-art by a large margin of +2.7 box AP and +2.6 mask AP on COCO, and +3.2 mIoU on ADE20K, demonstrating the potential of Transformer-based models as vision backbones. The hierarchical design and the shifted window approach also prove beneficial for all-MLP architectures. The code and models are publicly available at~\url{https://github.com/microsoft/Swin-Transformer}.

Motivation & Objective

• Develop a general-purpose Transformer backbone for vision that handles multi-scale visual entities.
• Achieve linear computational complexity with respect to image size via local windowed self-attention.
• Enable hierarchical feature maps to support dense prediction tasks with compatibility to FPN/U-Net styles.
• Bridge windows across layers through shifted window partitioning to enhance modeling power.

Proposed method

• Split images into patch tokens and linearly embed them to form hierarchical stages.
• Compute self-attention locally within non-overlapping windows to achieve linear complexity.
• Apply a shifted window strategy between consecutive blocks to enable cross-window connections.
• Use a relative position bias in self-attention to improve spatial modeling.
• Construct Swin Transformer blocks with W-MSA and SW-MSA followed by MLP with GELU and residual connections.
• Provide multiple model sizes (Swin-T/Swin-S/Swin-B/Swin-L) with specified stage configurations.

Figure 1: (a) The proposed Swin Transformer builds hierarchical feature maps by merging image patches (shown in gray) in deeper layers and has linear computation complexity to input image size due to computation of self-attention only within each local window (shown in red). It can thus serve as a g

Experimental results

Research questions

• RQ1 Can a hierarchical Transformer with shifted window self-attention serve as a general backbone for both classification and dense vision tasks?
• RQ2 Does shifting the window partition between layers provide cross-window connectivity with acceptable latency?
• RQ3 How does Swin Transformer perform on ImageNet-1K, COCO object detection/instance segmentation, and ADE20K semantic segmentation compared to state-of-the-art backbones?

Key findings

• Swin-T achieves 81.3% top-1 on ImageNet-1K with regular training and scales to 86.4%/87.3% for Swin-B/L with ImageNet-22K pre-training.
• On COCO test-dev, Swin-T/B-L surpasses previous state-of-the-art by up to +2.7 box AP and +2.6 mask AP.
• On ADE20K val, Swin-S/L achieve +5.3 mIoU and +3.2 mIoU over prior best models, respectively.
• Swin Transformer significantly outperforms DeiT and ResNeXt/ResNet backbones with similar latency across classification, detection, and segmentation tasks.
• The shifted window design provides substantial modeling gains with only modest latency overhead, and relative position bias improves performance across tasks.

Figure 2: An illustration of the shifted window approach for computing self-attention in the proposed Swin Transformer architecture. In layer $l$ (left), a regular window partitioning scheme is adopted, and self-attention is computed within each window. In the next layer $l+1$ (right), the window pa