[Paper Review] Residual Attention Network for Image Classification

Fei Wang, Mengqing Jiang+6 more|arXiv (Cornell University)|Apr 23, 2017

Advanced Neural Network ApplicationsComputer Science38 references315 citations

TL;DR

The paper introduces Residual Attention Network, stacking Attention Modules with bottom-up top-down mask branches and attention residual learning to achieve state-of-the-art image classification performance while enabling very deep networks. It demonstrates strong results on CIFAR-10/100 and ImageNet with reduced computation.

ABSTRACT

In this work, we propose "Residual Attention Network", a convolutional neural network using attention mechanism which can incorporate with state-of-art feed forward network architecture in an end-to-end training fashion. Our Residual Attention Network is built by stacking Attention Modules which generate attention-aware features. The attention-aware features from different modules change adaptively as layers going deeper. Inside each Attention Module, bottom-up top-down feedforward structure is used to unfold the feedforward and feedback attention process into a single feedforward process. Importantly, we propose attention residual learning to train very deep Residual Attention Networks which can be easily scaled up to hundreds of layers. Extensive analyses are conducted on CIFAR-10 and CIFAR-100 datasets to verify the effectiveness of every module mentioned above. Our Residual Attention Network achieves state-of-the-art object recognition performance on three benchmark datasets including CIFAR-10 (3.90% error), CIFAR-100 (20.45% error) and ImageNet (4.8% single model and single crop, top-5 error). Note that, our method achieves 0.6% top-1 accuracy improvement with 46% trunk depth and 69% forward FLOPs comparing to ResNet-200. The experiment also demonstrates that our network is robust against noisy labels.

Motivation & Objective

• Motivate and develop a deep convolutional network that integrates mixed attention mechanisms to improve feature discrimination.
• Introduce an attention module with bottom-up top-down mask branches to softly weight trunk features.
• Enable training of very deep networks via attention residual learning to mitigate degradation.
• Demonstrate robustness to noisy labels and improved parameter efficiency compared to ResNet and related architectures.

Proposed method

• Stack Attention Modules to form a Residual Attention Network where each module has a mask branch and a trunk branch.
• Use a bottom-up top-down feedforward structure in the mask branch to generate an attention mask M(x) in [0,1].
• Apply attention residual learning: H(x) = (1 + M(x)) * F(x) to preserve information and improve gradient flow.
• Experiment with mixed, channel, and spatial attention activations; select mixed attention as best performing.
• Replace trunk unit with ResNet/ResNeXt/Inception variants to show architectural compatibility and efficiency.
• Evaluate on CIFAR-10, CIFAR-100, and ImageNet; compare against ResNet, WRN, and other baselines.

Experimental results

Research questions

• RQ1 Can stacked Attention Modules with bottom-up top-down masks improve image classification beyond baseline deep networks?
• RQ2 Does attention residual learning enable training of hundreds of layers without degraded performance?
• RQ3 What is the impact of different attention activations (mixed, channel, spatial) on classification accuracy?
• RQ4 How does the Residual Attention Network perform on CIFAR-10/100 and ImageNet compared to state-of-the-art methods in terms of accuracy and efficiency?
• RQ5 Is the approach robust to noisy labels during training?

Key findings

• Attention Residual Learning consistently improves performance as the number of Attention Modules increases.
• Mixed attention activation yields the best accuracy among tested options on CIFAR-10.
• Attention-452 achieves 3.90% error on CIFAR-10 and 20.45% on CIFAR-100 with markedly fewer parameters and FLOPs than some ResNet baselines.
• On ImageNet, Attention-56 and Attention-92 outperform several ResNet/ResNeXt/Inception baselines under single-crop evaluation, with lower or comparable FLOPs and parameters.
• The network demonstrates robustness to noisy labels due to gradient filtering by the soft mask branch.