[Paper Review] SG-Former: Self-guided Transformer with Evolving Token Reallocation

Sucheng Ren, Xingyi Yang+2 more|arXiv (Cornell University)|Aug 23, 2023

Advanced Neural Network Applications4 citations

TL;DR

SG-Former introduces a self-guided Transformer with evolving token reallocation that adaptively allocates more tokens to salient image regions for fine-grained attention while reducing tokens in less important areas to maintain global context and efficiency. By using a significance map derived from hybrid-scale self-attention that evolves during training, it achieves state-of-the-art performance with lower FLOPs and fewer parameters, attaining 84.7% Top-1 accuracy on ImageNet-1K and surpassing Swin Transformer by +1.3% accuracy and +3 mIoU on ADE20K.

ABSTRACT

Vision Transformer has demonstrated impressive success across various vision tasks. However, its heavy computation cost, which grows quadratically with respect to the token sequence length, largely limits its power in handling large feature maps. To alleviate the computation cost, previous works rely on either fine-grained self-attentions restricted to local small regions, or global self-attentions but to shorten the sequence length resulting in coarse granularity. In this paper, we propose a novel model, termed as Self-guided Transformer~(SG-Former), towards effective global self-attention with adaptive fine granularity. At the heart of our approach is to utilize a significance map, which is estimated through hybrid-scale self-attention and evolves itself during training, to reallocate tokens based on the significance of each region. Intuitively, we assign more tokens to the salient regions for achieving fine-grained attention, while allocating fewer tokens to the minor regions in exchange for efficiency and global receptive fields. The proposed SG-Former achieves performance superior to state of the art: our base size model achieves extbf{84.7\%} Top-1 accuracy on ImageNet-1K, extbf{51.2mAP} bbAP on CoCo, extbf{52.7mIoU} on ADE20K surpassing the Swin Transformer by extbf{+1.3\% / +2.7 mAP/ +3 mIoU}, with lower computation costs and fewer parameters. The code is available at \href{https://github.com/OliverRensu/SG-Former}{https://github.com/OliverRensu/SG-Former}

Motivation & Objective

• To address the quadratic computation cost of Vision Transformers in high-resolution feature maps.
• To enable fine-grained global self-attention without sacrificing global receptive fields or increasing computational cost.
• To dynamically reallocate tokens based on region significance, improving representation quality in salient areas.
• To reduce dependency on hand-designed token aggregation strategies by learning significance maps end-to-end.
• To unify local and global attention within a single self-attention mechanism for adaptive granularity.

Proposed method

• The model uses hybrid-scale self-attention to extract multi-granularity features within a single layer, grouping attention heads to capture both local and global patterns.
• A significance map is estimated from the hybrid-scale attention output to identify salient image regions based on their importance.
• Tokens are reallocated based on the significance map: more tokens are assigned to salient regions for fine-grained interaction, fewer to background regions for efficiency.
• The significance map evolves during training, enabling self-guided, adaptive reallocation that is image-specific and data-driven.
• Key and value tokens are reallocated while query tokens remain fixed, enabling efficient global self-attention with dynamic token distribution.
• The method preserves long-range dependencies across the entire feature map while focusing computational resources on semantically important regions.

Experimental results

Research questions

• RQ1 Can a self-attention mechanism dynamically reallocate tokens to salient regions to improve fine-grained representation with global context?
• RQ2 How does hybrid-scale self-attention contribute to more accurate significance map estimation for token reallocation?
• RQ3 Does self-guided, evolving token reallocation outperform fixed or uniform token aggregation strategies in terms of accuracy and efficiency?
• RQ4 Can adaptive token allocation maintain global receptive fields while reducing computation cost in high-resolution feature maps?
• RQ5 What is the impact of using significance maps derived from local, global, or hybrid-scale attention on model performance?

Key findings

• SG-Former achieves 84.7% Top-1 accuracy on ImageNet-1K, surpassing Swin Transformer by +1.3%.
• On COCO object detection, it achieves 51.2 mAP bbAP, outperforming Swin Transformer by +2.7 mAP.
• On ADE20K semantic segmentation, it reaches 52.7 mIoU, exceeding Swin Transformer by +3 mIoU.
• With a base model, SG-Former reduces FLOPs and parameters compared to Swin Transformer while improving performance.
• The ablation study shows that hybrid-scale significance maps outperform local, global, or manually defined maps by +0.9 Top-1 accuracy.
• Using Semantic FPN, SG-Former achieves 50.6 mIoU on ADE20K, outperforming Swin by 4.6 mIoU with 10% fewer parameters.