[Paper Review] Interlaced Sparse Self-Attention for Semantic Segmentation

Lang Huang, Yuhui Yuan+4 more|arXiv (Cornell University)|Jul 29, 2019

Advanced Neural Network Applications60 references53 citations

TL;DR

The paper introduces Interlaced Sparse Self-Attention (IANet) to capture long-range context for semantic segmentation and related tasks, showing consistent improvements over baselines and non-local methods on Cityscapes, ADE20K, LIP, and COCO.

ABSTRACT

In this paper, we present a so-called interlaced sparse self-attention approach to improve the efficiency of the \emph{self-attention} mechanism for semantic segmentation. The main idea is that we factorize the dense affinity matrix as the product of two sparse affinity matrices. There are two successive attention modules each estimating a sparse affinity matrix. The first attention module is used to estimate the affinities within a subset of positions that have long spatial interval distances and the second attention module is used to estimate the affinities within a subset of positions that have short spatial interval distances. These two attention modules are designed so that each position is able to receive the information from all the other positions. In contrast to the original self-attention module, our approach decreases the computation and memory complexity substantially especially when processing high-resolution feature maps. We empirically verify the effectiveness of our approach on six challenging semantic segmentation benchmarks.

Motivation & Objective

• Motivate and improve semantic segmentation by effectively modeling long-range contextual dependencies.
• Propose an interlaced attention mechanism that alternates long-range and short-range context aggregation.
• Demonstrate generalization across semantic segmentation, object detection, and instance segmentation tasks.
• Compare with baseline, non-local, and related attention methods through extensive ablations and benchmarks.

Proposed method

• Propose Interlaced Sparse Self-Attention (IANet) that cascades long-range and short-range attention blocks.
• Replace or augment self-attention blocks in backbone networks with interlaced attention to capture global context.
• Conduct ablations comparing IA against baseline, NL, RCCA, and CGNL across multiple datasets.
• Evaluate on Cityscapes, ADE20K, and LIP for segmentation, and COCO with Mask-RCNN for detection/instance segmentation.
• Use ImageNet-pretrained backbones with dilated convolutions and auxiliary loss; employ poly learning rate policy and synchronized batch normalization.

Experimental results

Research questions

• RQ1 Does interlaced attention improve segmentation performance over baseline and non-local/self-attention methods across diverse datasets?
• RQ2 How does the interlaced attention compare with other context modeling approaches (NL, RCCA, CGNL) in terms of accuracy and efficiency?
• RQ3 What is the impact of partitioning (L) and the order of long-range vs short-range attention on performance?
• RQ4 Can the proposed IA technique generalize to object detection and instance segmentation tasks like COCO with Mask-RCNN?
• RQ5 How does adding multiple IA blocks influence performance in other tasks (e.g., CUB-200-2011 classification)?

Key findings

• Interlaced attention provides significant improvements over the baseline and non-local methods in semantic segmentation benchmarks.
• IANet achieves state-of-the-art or competitive results on Cityscapes, ADE20K, and LIP compared to prior methods using similar backbones.
• Adding a single interlaced attention block yields consistent gains in COCO object detection and instance segmentation over Mask-RCNN baseline.
• Compared with CGNL and NL, IA delivers better Top-1/Top-5 accuracy on CUB-200-2011 and outperforms RCCA in Cityscapes ablations.
• Partition size and the order of attention stages affect performance, with larger partitions and Long-Range then Short-Range attention performing best.