[Paper Review] AttaNet: Attention-Augmented Network for Fast and Accurate Scene Parsing

Qi Song, Kangfu Mei+1 more|arXiv (Cornell University)|Mar 10, 2021

Advanced Neural Network Applications41 references4 citations

TL;DR

AttaNet is a lightweight, efficient neural network for real-time scene parsing that introduces the Strip Attention Module (SAM) to capture global context with reduced computational cost by focusing on vertical and horizontal strip-based attention, and the Attention Fusion Module (AFM) to intelligently combine multi-level features. It achieves state-of-the-art speed-accuracy trade-offs, reaching 180 FPS with 70.1% mIoU on Cityscapes and 43.71% mIoU on ADE20K with significantly lower FLOPs than prior methods.

ABSTRACT

Two factors have proven to be very important to the performance of semantic segmentation models: global context and multi-level semantics. However, generating features that capture both factors always leads to high computational complexity, which is problematic in real-time scenarios. In this paper, we propose a new model, called Attention-Augmented Network (AttaNet), to capture both global context and multilevel semantics while keeping the efficiency high. AttaNet consists of two primary modules: Strip Attention Module (SAM) and Attention Fusion Module (AFM). Viewing that in challenging images with low segmentation accuracy, there are a significantly larger amount of vertical strip areas than horizontal ones, SAM utilizes a striping operation to reduce the complexity of encoding global context in the vertical direction drastically while keeping most of contextual information, compared to the non-local approaches. Moreover, AFM follows a cross-level aggregation strategy to limit the computation, and adopts an attention strategy to weight the importance of different levels of features at each pixel when fusing them, obtaining an efficient multi-level representation. We have conducted extensive experiments on two semantic segmentation benchmarks, and our network achieves different levels of speed/accuracy trade-offs on Cityscapes, e.g., 71 FPS/79.9% mIoU, 130 FPS/78.5% mIoU, and 180 FPS/70.1% mIoU, and leading performance on ADE20K as well.

Motivation & Objective

• To address the high computational cost of existing models that use global context and multi-level semantics for semantic segmentation.
• To enable real-time inference without sacrificing segmentation accuracy by reducing FLOPs in attention mechanisms.
• To develop an efficient module that captures long-range dependencies using directional striping instead of full self-attention.
• To fuse multi-level features effectively using an attention-based strategy that weights feature importance per pixel.
• To ensure compatibility and performance gains across diverse backbone networks, from lightweight to deep architectures.

Proposed method

• Introduces the Strip Attention Module (SAM) that applies vertical and horizontal striping operations to reduce the spatial complexity of self-attention from O(H×W×H×W) to O(H×W×W), focusing on directional long-range dependencies.
• Uses a striping operation to encode global context along columns (vertical) or rows (horizontal), reducing computation while preserving structural consistency in banded objects like fences and poles.
• Proposes the Attention Fusion Module (AFM) to perform cross-level feature aggregation with learnable attention weights, allowing each pixel to dynamically select relevant features from different levels.
• Applies AFM only between the last two stages of the backbone to minimize computational overhead while maximizing feature representation quality.
• Designs the network to be plug-and-play, enabling integration with various backbone networks including ResNet50, ResNet101, ShuffleNetV2, DF1, and DF2.
• Employs a cross-level aggregation architecture with minimal additional parameters and FLOPs, ensuring high efficiency without compromising performance.

Experimental results

Research questions

• RQ1 Can a more efficient attention mechanism be designed to reduce the computational cost of global context modeling in semantic segmentation?
• RQ2 How can multi-level features be fused effectively without introducing significant computational overhead?
• RQ3 Can directional striping operations improve long-range dependency modeling while preserving structural details in segmentation?
• RQ4 Does the proposed attention-based fusion strategy outperform simple concatenation or element-wise summation in multi-level feature integration?
• RQ5 Can the proposed modules be generalized across different backbone architectures, including lightweight and deep networks?

Key findings

• AttaNet achieves 180 FPS with 70.1% mIoU on Cityscapes, demonstrating a strong speed-accuracy trade-off.
• On Cityscapes, AttaNet reaches 130 FPS with 78.5% mIoU and 71 FPS with 79.9% mIoU, outperforming many existing methods in both speed and accuracy.
• With ResNet50, AttaNet improves mIoU from 73.4% to 81.2% while increasing FLOPs by only 3.2%, showing high efficiency.
• On ADE20K, AttaNet achieves 43.71% mIoU with only 150.5 GFLOPs, significantly outperforming models with higher FLOPs.
• SAM reduces FLOPs by 94.5% compared to non-local blocks and 60.8% compared to CCNet’s RCCA module, with comparable or better segmentation quality.
• AFM achieves the best performance among fusion methods with only a marginal increase in computation, proving effective for multi-level feature integration.