Skip to main content
QUICK REVIEW

[Paper Review] MaxViT: Multi-Axis Vision Transformer

Zhengzhong Tu, Hossein Talebi|arXiv (Cornell University)|Apr 4, 2022
Visual Attention and Saliency DetectionComputer Science26 citations
TL;DR

MaxViT introduces a multi-axis self-attention mechanism that blends blocked local attention with dilated global attention and combines it with convolutions to form a scalable, hierarchical vision backbone, achieving state-of-the-art results on ImageNet and COCO.

ABSTRACT

Transformers have recently gained significant attention in the computer vision community. However, the lack of scalability of self-attention mechanisms with respect to image size has limited their wide adoption in state-of-the-art vision backbones. In this paper we introduce an efficient and scalable attention model we call multi-axis attention, which consists of two aspects: blocked local and dilated global attention. These design choices allow global-local spatial interactions on arbitrary input resolutions with only linear complexity. We also present a new architectural element by effectively blending our proposed attention model with convolutions, and accordingly propose a simple hierarchical vision backbone, dubbed MaxViT, by simply repeating the basic building block over multiple stages. Notably, MaxViT is able to ''see'' globally throughout the entire network, even in earlier, high-resolution stages. We demonstrate the effectiveness of our model on a broad spectrum of vision tasks. On image classification, MaxViT achieves state-of-the-art performance under various settings: without extra data, MaxViT attains 86.5% ImageNet-1K top-1 accuracy; with ImageNet-21K pre-training, our model achieves 88.7% top-1 accuracy. For downstream tasks, MaxViT as a backbone delivers favorable performance on object detection as well as visual aesthetic assessment. We also show that our proposed model expresses strong generative modeling capability on ImageNet, demonstrating the superior potential of MaxViT blocks as a universal vision module. The source code and trained models will be available at https://github.com/google-research/maxvit.

Motivation & Objective

  • Motivate scalable vision architectures that capture both local and global interactions.
  • Develop a transformer block that combines block-local and grid-global attention with convolutions.
  • Build a simple hierarchical backbone by repeating the MaxViT block across stages.
  • Demonstrate strong performance across classification, detection, aesthetics, and generation tasks.

Proposed method

  • Introduce multi-axis self-attention (Max-SA) that decomposes full attention into block-local and grid-global attention with linear complexity.
  • Blend Max-SA with MBConv blocks and SE for improved generalization, using MBConv as a conditional position encoding.
  • Construct a hierarchical backbone MaxViT by stacking repeated MaxViT blocks across four stages (S0-S4).
  • Provide configurable variants (MaxViT-T, -S, -B, -L, -XL) with varying blocks per stage and channel sizes.
  • Show that Max-SA can drop in as a Swin-like attention drop-in with same parameter count and FLOPs, but with global interactions at all stages.

Experimental results

Research questions

  • RQ1Can multi-axis attention (local block + global grid) provide global context with linear complexity in high-resolution vision tasks?
  • RQ2Does combining Max-SA with convolutions in a simple hierarchical backbone improve accuracy and efficiency over existing Vision Transformers and hybrids?
  • RQ3How do various architectural choices (block order, sequential vs parallel attention, inclusion of MBConv, vertical layout) affect performance across vision tasks?
  • RQ4To what extent does MaxViT scale with data (ImageNet-1K, ImageNet-21K, JFT-300M) and across downstream tasks like detection and aesthetics?
  • RQ5Is MaxViT capable of strong generative performance in image generation settings?

Key findings

  • MaxViT achieves state-of-the-art ImageNet-1K top-1 accuracy across settings, including 86.7% with 512x512 fine-tuning for MaxViT-L and 85.17% for MaxViT-L at 224x224 without extra data.
  • With ImageNet-21K pre-training, MaxViT-B reaches 88.38% top-1 accuracy and MaxViT-XL reaches 88.70% at 512x512, outperforming prior models at similar or larger sizes.
  • On JFT-300M-scale data, MaxViT-XL achieves 89.53% top-1 accuracy, demonstrating strong scaling to massive datasets.
  • In COCO object detection/instance segmentation, MaxViT backbones outperform Swin, ConvNeXt, and UViT at various sizes, with notable margins at base levels (e.g., MaxViT-S outperforming Swin-B and UViT-B at similar FLOPs).
  • On image aesthetics (AVA), MaxViT-T shows competitive PLCC/SRCC scores and improves with higher resolutions compared to prior methods.
  • In unconditional 128x128 image generation, MaxViT yields better FID/IS with fewer parameters than HiT and other baselines.

Read this paper

Open the full text in the Nubint reader and ask about any passage as you read.

Read this paper on Nubint

This review was generated by AI from the original paper. Please check the paper before citing it.