[Paper Review] Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles

Chaitanya K. Ryali, Yuan-Ting Hu+11 more|arXiv (Cornell University)|Jun 1, 2023

CCD and CMOS Imaging Sensors58 citations

TL;DR

Hiera is a simple hierarchical vision transformer pretrained with MAE that removes conventional vision-specific modules and achieves faster training/inference with higher accuracy on image and video tasks compared to prior hierarchical models.

ABSTRACT

Modern hierarchical vision transformers have added several vision-specific components in the pursuit of supervised classification performance. While these components lead to effective accuracies and attractive FLOP counts, the added complexity actually makes these transformers slower than their vanilla ViT counterparts. In this paper, we argue that this additional bulk is unnecessary. By pretraining with a strong visual pretext task (MAE), we can strip out all the bells-and-whistles from a state-of-the-art multi-stage vision transformer without losing accuracy. In the process, we create Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training. We evaluate Hiera on a variety of tasks for image and video recognition. Our code and models are available at https://github.com/facebookresearch/hiera.

Motivation & Objective

• Motivate that added vision-specific modules in hierarchical transformers add bulk and slow down models.
• Show that strong MAE pretraining can teach spatial biases, enabling a simpler architecture to outperform complex predecessors.
• Demonstrate that a streamlined hierarchical ViT can be faster in training and inference while maintaining or improving accuracy across image and video tasks.

Proposed method

• Start from a base hierarchical ViT (MViTv2) and strip non-essential vision-specific components.
• Adopt Masked Autoencoder (MAE) pretraining with a custom notion of mask units to suit hierarchical models.
• Introduce Mask Unit Attention to replace global KV pooling in early stages for efficiency during MAE pretraining.
• Use a separate-and-pad trick to avoid overlap issues during sparse MAE pretraining, enabling efficient training without convolutions.
• Remove relative position embeddings, convolutions, and many bells-and-whistles while maintaining performance under MAE pretraining.
• Provide multiple Hiera configurations (T, S, B, B+, L, H) with detailed FLOPs, parameters, and stage structures.

Experimental results

Research questions

• RQ1 Can a purely transformer-based, hierarchical vision model achieve state-of-the-art results without bells-and-whistles when pretrained with MAE?
• RQ2 How does MAE pretraining interact with a simplified hierarchical architecture to deliver speed and accuracy gains in image and video tasks?
• RQ3 What are the trade-offs in removing convolutions, attention residuals, and specialized pooling when using MAE on hierarchical ViTs?
• RQ4 How do different MAE pretraining settings (mask ratio, decoder depth, drop path, pretraining length) affect performance on images vs. video?

Key findings

• Hiera outperforms previous state-of-the-art hierarchical models on image and video tasks while being significantly faster in training and inference.
• MAE pretraining enables a simple architecture to achieve competitive or superior accuracy compared to more complex, convolution-heavy models.
• Removing convolutions, shifted windows, and decomposed relative position embeddings while using MAE yields an extremely simple model with strong performance.
• Mask Unit Attention provides a substantial throughput boost in video without hurting accuracy, contributing to large speedups over the base MViTv2 architecture.
• Across variants, Hiera-L achieves 86.1% top-1 on ImageNet-1K with MAE and 88.3% on Kinetics-700 in video, while being smaller and faster than comparable baselines.
• On transfer tasks, Hiera variants consistently outperform ViT pretrained with MAE on iNaturalist and Places datasets.