[Paper Review] XCiT: Cross-Covariance Image Transformers

Alaaeldin El-Nouby, Hugo Touvron+9 more|arXiv (Cornell University)|Jun 17, 2021

Domain Adaptation and Few-Shot Learning81 references233 citations

TL;DR

XCiT introduces cross-covariance attention (XCA) that operates across feature channels with linear complexity, enabling scalable vision transformers that match or exceed baselines on ImageNet, COCO, ADE20k, and self-supervised learning benchmarks.

ABSTRACT

Following their success in natural language processing, transformers have recently shown much promise for computer vision. The self-attention operation underlying transformers yields global interactions between all tokens ,i.e. words or image patches, and enables flexible modelling of image data beyond the local interactions of convolutions. This flexibility, however, comes with a quadratic complexity in time and memory, hindering application to long sequences and high-resolution images. We propose a "transposed" version of self-attention that operates across feature channels rather than tokens, where the interactions are based on the cross-covariance matrix between keys and queries. The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images. Our cross-covariance image transformer (XCiT) is built upon XCA. It combines the accuracy of conventional transformers with the scalability of convolutional architectures. We validate the effectiveness and generality of XCiT by reporting excellent results on multiple vision benchmarks, including image classification and self-supervised feature learning on ImageNet-1k, object detection and instance segmentation on COCO, and semantic segmentation on ADE20k.

Motivation & Objective

• Motivate efficient, scalable transformers for high-resolution vision tasks.
• Replace standard self-attention with cross-covariance attention to reduce complexity from quadratic to linear in the number of tokens.
• Combine XCA with local patch interaction and feed-forward networks to build practical vision transformers.
• Demonstrate XCiT performance across image classification, object detection, semantic segmentation, and self-supervised learning.

Proposed method

• Define cross-covariance attention (XCA) operating on feature dimensions rather than token dimensions.
• Apply row/column normalization and a learnable temperature parameter to stabilize training.
• Introduce block-diagonal multi-head structure to reduce complexity and ease optimization.
• Integrate XCA with Local Patch Interaction (LPI) and a pointwise feed-forward network in a columnar encoder architecture.
• Use class attention for global aggregation in image classification tasks.
• Experiment with patch sizes (8x8 and 16x16) and resolutions up to 384x384, including distillation with convolutional teachers.

Experimental results

Research questions

• RQ1 Can cross-covariance attention provide a scalable alternative to quadratic self-attention for high-resolution vision tasks?
• RQ2 Does XCA maintain accuracy while reducing computational and memory costs across image classification, detection, and segmentation benchmarks?
• RQ3 How does XCiT perform in self-supervised learning setups compared to ViT and convolutional backbones?
• RQ4 Does a block-diagonal (grouped) attention structure improve optimization and performance?
• RQ5 Is XCiT robust to varying input resolutions during training and testing?

Key findings

• XCiT-L24/16 achieves 86.0% top-1 accuracy on ImageNet-1k at 224x224 with 189M parameters and 417.9 GFLOPs.
• On COCO, XCiT backbones reach 48.5% AP for object detection and 43.7% AP for instance segmentation.
• On ADE20k semantic segmentation, XCiT reaches 48.4% mIoU, outperforming Swin Transformer backbones of comparable size.
• In self-supervised learning with DINO, XCiT attains 80.9% top-1 accuracy on ImageNet-1k.
• XCA operates with linear complexity in the number of tokens, enabling efficient processing of high-resolution images.
• Class-attention visualizations show heads focusing on semantically coherent regions and adapting to content.