[Paper Review] Focal Modulation Networks

Jianwei Yang, Chunyuan Li+3 more|arXiv (Cornell University)|Mar 22, 2022

Advanced Neural Network ApplicationsComputer Science148 citations

TL;DR

Focal Modulation Networks replace self-attention with a focal modulation module that aggregates multi-scale context via depth-wise convolutions and modulates per-token queries, achieving state-of-the-art results on classification, detection, and segmentation with competitive efficiency.

ABSTRACT

We propose focal modulation networks (FocalNets in short), where self-attention (SA) is completely replaced by a focal modulation mechanism for modeling token interactions in vision. Focal modulation comprises three components: (i) hierarchical contextualization, implemented using a stack of depth-wise convolutional layers, to encode visual contexts from short to long ranges, (ii) gated aggregation to selectively gather contexts for each query token based on its content, and (iii) element-wise modulation or affine transformation to inject the aggregated context into the query. Extensive experiments show FocalNets outperform the state-of-the-art SA counterparts (e.g., Swin and Focal Transformers) with similar computational costs on the tasks of image classification, object detection, and segmentation. Specifically, FocalNets with tiny and base size achieve 82.3% and 83.9% top-1 accuracy on ImageNet-1K. After pretrained on ImageNet-22K in 224 resolution, it attains 86.5% and 87.3% top-1 accuracy when finetuned with resolution 224 and 384, respectively. When transferred to downstream tasks, FocalNets exhibit clear superiority. For object detection with Mask R-CNN, FocalNet base trained with 1 imes outperforms the Swin counterpart by 2.1 points and already surpasses Swin trained with 3 imes schedule (49.0 v.s. 48.5). For semantic segmentation with UPerNet, FocalNet base at single-scale outperforms Swin by 2.4, and beats Swin at multi-scale (50.5 v.s. 49.7). Using large FocalNet and Mask2former, we achieve 58.5 mIoU for ADE20K semantic segmentation, and 57.9 PQ for COCO Panoptic Segmentation. Using huge FocalNet and DINO, we achieved 64.3 and 64.4 mAP on COCO minival and test-dev, respectively, establishing new SoTA on top of much larger attention-based models like Swinv2-G and BEIT-3. Code and checkpoints are available at https://github.com/microsoft/FocalNet.

Motivation & Objective

• Develop an attention-free mechanism to model input-dependent long-range interactions in vision tasks.
• Capture short- and long-range visual context using a multi-level hierarchical modulation.
• Demonstrate accuracy and efficiency gains over state-of-the-art self-attention based models across classification, detection, and segmentation.

Proposed method

• Replace self-attention with a Focal Modulation module that first aggregates context at multiple focal levels with depth-wise convolutions.
• Compute a modulator from aggregated context via a gated aggregation mechanism, then inject it into the query through an element-wise affine-like interaction.
• Use a two-step context aggregation: (i) hierarchical contextualization with a stack of depth-wise convolutions, and (ii) gated aggregation to form a per-token modulator.
• Define y_i = q(x_i) ⊙ h(Z_out) where Z_out encodes multi-level context and gating weights select the contribution of each level.
• Incorporate non-linearities (GeLU) and design choices that preserve translation invariance and explicit input-dependency for modulation.
• Discuss complexity: primarily governed by 3C^2 + C(2L+3) + C∑(k^ℓ)^2, with efficient, attention-free token interactions.

Experimental results

Research questions

• RQ1 Can an attention-free modulation mechanism match or surpass self-attention in vision models across classification, detection, and segmentation?
• RQ2 Does multi-scale contextual aggregation followed by per-token modulation improve accuracy while maintaining or improving computational efficiency?
• RQ3 How does Focal Modulation compare to Swin and Focal Transformers in dense prediction tasks and large-scale pretraining settings?
• RQ4 What qualitative interpretability benefits arise from focal modulation compared to traditional attention mechanisms?

Key findings

• FocalNets outperform state-of-the-art self-attention based counterparts with comparable costs across image classification, object detection, and segmentation.
• Tiny and base FocalNets achieve 82.3% and 83.9% top-1 on ImageNet-1K, with further gains when pretrained on ImageNet-22K (86.5%/87.3% top-1 at 224^2/384^2 finetuning).
• On COCO object detection, FocalNet base trained with 1× schedule surpasses Swin with 3× and shows competitive results across detection variants.
• For ADE20K segmentation, FocalNet base achieves 50.5 mIoU single-scale, outperforming Swin multi-scale (49.7); large FocalNet reaches 58.5 mIoU and 57.9 PQ on ADE20K and COCO Panoptic, respectively.
• When paired with larger backbones and training regimes, FocalNets establish new SOTA on COCO minival/test-dev for mAP with models like Swinv2-G and BEIT-3.
• Visualizations indicate the modulators focus on object regions that induce recognition categories, highlighting interpretability of FocalNets.