[Paper Review] Dual Attention Network for Scene Segmentation

Jun Fu, Jing Liu+5 more|arXiv (Cornell University)|Sep 9, 2018

Advanced Neural Network ApplicationsComputer Science38 references211 citations

TL;DR

DANet introduces spatial (position) and channel attention modules to capture global dependencies on top of a dilated FCN, achieving state-of-the-art results on Cityscapes, PASCAL Context, COCO Stuff, and PASCAL VOC 2012 datasets.

ABSTRACT

In this paper, we address the scene segmentation task by capturing rich contextual dependencies based on the selfattention mechanism. Unlike previous works that capture contexts by multi-scale features fusion, we propose a Dual Attention Networks (DANet) to adaptively integrate local features with their global dependencies. Specifically, we append two types of attention modules on top of traditional dilated FCN, which model the semantic interdependencies in spatial and channel dimensions respectively. The position attention module selectively aggregates the features at each position by a weighted sum of the features at all positions. Similar features would be related to each other regardless of their distances. Meanwhile, the channel attention module selectively emphasizes interdependent channel maps by integrating associated features among all channel maps. We sum the outputs of the two attention modules to further improve feature representation which contributes to more precise segmentation results. We achieve new state-of-the-art segmentation performance on three challenging scene segmentation datasets, i.e., Cityscapes, PASCAL Context and COCO Stuff dataset. In particular, a Mean IoU score of 81.5% on Cityscapes test set is achieved without using coarse data. We make the code and trained model publicly available at https://github.com/junfu1115/DANet

Motivation & Objective

• Motivate improved scene segmentation by modeling long-range contextual dependencies beyond multi-scale feature fusion.
• Propose two complementary self-attention modules (position and channel) to capture spatial and inter-channel relationships.
• Demonstrate that fusing attention module outputs enhances feature representations for pixel-level prediction.

Proposed method

• Add two parallel self-attention modules on top of a dilated FCN backbone.
• Position Attention Module: compute a spatial attention map S and produce E = alpha * D * S^T + A, with alpha learned starting from 0.
• Channel Attention Module: compute a channel attention map X from A and produce E = beta * X * A^T + A, with beta learned starting from 0.
• Fuse the two attention-enhanced features via a convolutional embedding and element-wise sum, followed by a final convolution to produce the prediction map.

Experimental results

Research questions

• RQ1 Can self-attention modeling of spatial relations (position attention) improve pixel-wise segmentation by capturing long-range dependencies?
• RQ2 Can modeling interdependencies between feature channels (channel attention) improve discriminability of semantic classes?
• RQ3 Does combining both spatial and channel attention yield better performance than either module alone?

Key findings

• Attending with position attention alone improves Mean IoU to 75.74% (ResNet-50 baseline).
• Attending with channel attention alone improves Mean IoU by 4.25% over baseline.
• Combining both PAM and CAM yields 76.34% Mean IoU (ResNet-50).
• Using ResNet-101 as backbone with both modules achieves 77.57% Mean IoU on Cityscapes val.
• With data augmentation, multi-grid, and multi-scale inference, DANet-101 reaches 81.50% Mean IoU on Cityscapes val, outperforming Deeplabv3 and others in the table.