[Paper Review] On the Connection between Local Attention and Dynamic Depth-wise Convolution

Qi Han, Zejia Fan+5 more|arXiv (Cornell University)|Jun 8, 2021

CCD and CMOS Imaging SensorsEngineering70 citations

TL;DR

The paper analyzes local attention as a channel-wise, locally connected layer with dynamic weights and shows its connection to depth-wise convolution and its dynamic variants, supported by empirical ablations and comparisons across vision tasks.

ABSTRACT

Vision Transformer (ViT) attains state-of-the-art performance in visual recognition, and the variant, Local Vision Transformer, makes further improvements. The major component in Local Vision Transformer, local attention, performs the attention separately over small local windows. We rephrase local attention as a channel-wise locally-connected layer and analyze it from two network regularization manners, sparse connectivity and weight sharing, as well as weight computation. Sparse connectivity: there is no connection across channels, and each position is connected to the positions within a small local window. Weight sharing: the connection weights for one position are shared across channels or within each group of channels. Dynamic weight: the connection weights are dynamically predicted according to each image instance. We point out that local attention resembles depth-wise convolution and its dynamic version in sparse connectivity. The main difference lies in weight sharing - depth-wise convolution shares connection weights (kernel weights) across spatial positions. We empirically observe that the models based on depth-wise convolution and the dynamic variant with lower computation complexity perform on-par with or sometimes slightly better than Swin Transformer, an instance of Local Vision Transformer, for ImageNet classification, COCO object detection and ADE semantic segmentation. These observations suggest that Local Vision Transformer takes advantage of two regularization forms and dynamic weight to increase the network capacity. Code is available at https://github.com/Atten4Vis/DemystifyLocalViT.

Motivation & Objective

• Motivate understanding of local attention through network regularization lenses (sparse connectivity and weight sharing) and dynamic weight computation.
• Reformulate local attention as a channel-wise locally-connected layer with dynamic weights.
• Investigate the connection between local attention and (dynamic) depth-wise convolution both theoretically and empirically.
• Evaluate depth-wise convolution-based networks (DWNet) against local attention-based Swin Transformer in ImageNet, COCO, and ADE20K.
• Provide practical insights on weight sharing and dynamic weight mechanisms for improved efficiency and performance.

Proposed method

• Rephrase local attention as a channel-wise spatially-locally connected layer with dynamic weights.
• Compare sparse connectivity, weight sharing patterns, and dynamic weight computation between local attention and depth-wise convolution.
• Propose DWNet by replacing local attention in Swin Transformer with depth-wise convolution under the same architecture and window settings.
• Introduce homogeneous and inhomogeneous dynamic variants of depth-wise convolution with global pool-based or center-position-based weight prediction.
• Conduct ablation studies to assess the impact of weight sharing, dynamic weight, and window sampling strategies.
• Benchmark on ImageNet, COCO, and ADE20K under Swin Transformer-like training protocols.

Experimental results

Research questions

• RQ1 How does local attention relate to depth-wise convolution in terms of connectivity, weight sharing, and dynamic weight computation?
• RQ2 Do dynamic depth-wise convolution variants (DWNet) achieve competitive performance with local attention across ImageNet classification, COCO object detection, and ADE semantic segmentation?
• RQ3 What roles do weight sharing and dynamic weight prediction play in the effectiveness of local attention versus depth-wise convolution?
• RQ4 Can a depth-wise convolution-based architecture (DWNet) match or exceed Swin Transformer under comparable training settings?

Key findings

• Local attention is a channel-wise locally-connected layer with dynamic weights, sharing weights across channels and coupling via per-instance weight prediction.
• Depth-wise convolution shares weights across spatial positions and benefits from weight sharing across channels and/or positions, with dynamic variants using linear projections or center-based predictions.
• DWNet and its dynamic variants achieve comparable or slightly higher performance than Swin Transformer on ImageNet, COCO, and ADE semantic segmentation, with lower computational cost in several settings.
• Weight sharing across channels helps reduce parameter counts for local attention, while sharing across positions aids depth-wise convolution in reducing parameters and enabling translation-equivalent representations.
• Dynamic weight mechanisms improve performance for both local attention and depth-wise convolution, with linear-projection-based dynamic weights often favored over attention-based schemes in certain setups.