[Paper Review] CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention

Wenxiao Wang, Yao Lu+5 more|arXiv (Cornell University)|Jul 31, 2021

Advanced Neural Network Applications85 citations

TL;DR

CrossFormer introduces cross-scale embeddings and long-short distance attention to enable cross-scale interactions in vision transformers, plus a dynamic position bias for variable input sizes, achieving strong results across classification, detection, instance segmentation, and semantic segmentation.

ABSTRACT

Transformers have made great progress in dealing with computer vision tasks. However, existing vision transformers do not yet possess the ability of building the interactions among features of different scales, which is perceptually important to visual inputs. The reasons are two-fold: (1) Input embeddings of each layer are equal-scale, so no cross-scale feature can be extracted; (2) to lower the computational cost, some vision transformers merge adjacent embeddings inside the self-attention module, thus sacrificing small-scale (fine-grained) features of the embeddings and also disabling the cross-scale interactions. To this end, we propose Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA). On the one hand, CEL blends each embedding with multiple patches of different scales, providing the self-attention module itself with cross-scale features. On the other hand, LSDA splits the self-attention module into a short-distance one and a long-distance counterpart, which not only reduces the computational burden but also keeps both small-scale and large-scale features in the embeddings. Through the above two designs, we achieve cross-scale attention. Besides, we put forward a dynamic position bias for vision transformers to make the popular relative position bias apply to variable-sized images. Hinging on the cross-scale attention module, we construct a versatile vision architecture, dubbed CrossFormer, which accommodates variable-sized inputs. Extensive experiments show that CrossFormer outperforms the other vision transformers on image classification, object detection, instance segmentation, and semantic segmentation tasks. The code has been released: https://github.com/cheerss/CrossFormer.

Motivation & Objective

• Motivate the need for cross-scale feature interactions in vision transformers for diverse visual tasks.
• Propose mechanisms to enable cross-scale attention without prohibitive computation.
• Develop a versatile architecture that handles variable-sized inputs across tasks.

Proposed method

• Cross-scale Embedding Layer (CEL) samples patches at multiple scales per embedding and concatenates them to form cross-scale embeddings.
• Long Short Distance Attention (LSDA) splits self-attention into Short Distance Attention (SDA) and Long Distance Attention (LDA) to preserve both small-scale and large-scale features while reducing compute.
• Dynamic Position Bias (DPB) uses an MLP to generate relative position biases from embedding distances, enabling variable image/group sizes.
• A pyramid architecture with CEL and CrossFormer blocks (LSDA + DPB) is built to support multiple stages and diverse vision tasks.
• Four CrossFormer variants (T, S, B, L) are constructed to cover small to large model scales and different downstream needs.

Experimental results

Research questions

• RQ1 Can cross-scale embeddings and long-short distance attention enable effective cross-scale interactions in vision transformers?
• RQ2 Does a dynamic position bias improve robustness to variable input sizes and groupings across tasks?
• RQ3 Do CrossFormer models improve accuracy and task performance (classification, detection, instance/semantic segmentation) over state-of-the-art vision transformers?
• RQ4 How do CEL and LSDA contribute to computational efficiency without sacrificing small-scale detail?
• RQ5 Are the gains more pronounced on dense prediction tasks (detection/segmentation) than on image classification?

Key findings

• CrossFormer consistently outperforms state-of-the-art vision transformers on ImageNet classification with comparable parameters and FLOPs.
• CrossFormer-S/B/L achieve higher accuracy than strong baselines (e.g., DeiT, Swin) across model sizes, with notable gains on larger models.
• On COCO object detection and instance segmentation, CrossFormer variants attain higher AP metrics and show larger gains for bigger models.
• Semantic segmentation on ADE20K with CrossFormer backbones yields improved IOU/MS IOU scores, especially for larger variants.
• Ablations show cross-scale embeddings and LSDA outperform PVT-like and Swin-like attentions, and DPB matches RPB performance with added flexibility for variable input sizes.