[Paper Review] Learn To Pay Attention

Saumya Jetley, Nicholas A. Lord+2 more|arXiv (Cornell University)|Apr 6, 2018

Adversarial Robustness in Machine Learning35 references124 citations

TL;DR

An end-to-end trainable attention module for CNNs that uses a global image descriptor as a query to compute multi-scale, location-based attention; improves classification and weakly supervised segmentation, and offers some adversarial robustness gains.

ABSTRACT

We propose an end-to-end-trainable attention module for convolutional neural network (CNN) architectures built for image classification. The module takes as input the 2D feature vector maps which form the intermediate representations of the input image at different stages in the CNN pipeline, and outputs a 2D matrix of scores for each map. Standard CNN architectures are modified through the incorporation of this module, and trained under the constraint that a convex combination of the intermediate 2D feature vectors, as parameterised by the score matrices, must extit{alone} be used for classification. Incentivised to amplify the relevant and suppress the irrelevant or misleading, the scores thus assume the role of attention values. Our experimental observations provide clear evidence to this effect: the learned attention maps neatly highlight the regions of interest while suppressing background clutter. Consequently, the proposed function is able to bootstrap standard CNN architectures for the task of image classification, demonstrating superior generalisation over 6 unseen benchmark datasets. When binarised, our attention maps outperform other CNN-based attention maps, traditional saliency maps, and top object proposals for weakly supervised segmentation as demonstrated on the Object Discovery dataset. We also demonstrate improved robustness against the fast gradient sign method of adversarial attack.

Motivation & Objective

• Motivate and design an integrated attention mechanism that highlights salient image regions to improve CNN classification.
• Enable classification to be performed using a convex combination of local feature vectors guided by learned attention scores.
• Demonstrate that multi-scale attention can be added to existing architectures (e.g., VGG, ResNet) with performance gains on diverse datasets.
• Explore the attention maps’ usefulness for weakly supervised segmentation and adversarial robustness.
• Assess cross-domain generalization to unseen datasets.

Proposed method

• Define local feature vectors at intermediate layers and a global feature vector g.
• Compute compatibility scores between local features and g via a learnable compatibility function C.
• Normalize scores with softmax to obtain attention weights and form an attention-weighted global descriptor ga.
• Replace the original global descriptor with ga for final classification, enabling end-to-end training with cross-entropy loss.
• Investigate multiple configurations: single/multiple layers, dot-product vs parametrised compatibility, and concatenation or independent classifiers across layers.
• Apply attention to VGG and ResNet architectures and evaluate on CIFAR-10/100, CUB-200-2011, SVHN, and cross-domain datasets; also assess weakly supervised segmentation and adversarial robustness.

Experimental results

Research questions

• RQ1 Does incorporating an end-to-end trainable attention module improve image classification performance on standard and fine-grained datasets?
• RQ2 Can attention weighted representations enhance generalization to domain-shifted data?
• RQ3 Are attention maps effective for weakly supervised segmentation without pixel-level annotations?
• RQ4 How does the proposed attention mechanism affect robustness to adversarial perturbations?
• RQ5 What is the effect of multi-scale attention (across layers) on recognition across object parts and whole objects?

Key findings

• Attention-augmented networks outperform baselines on CIFAR-10/100, CIFAR/CUB/SVHN fine-grained tasks, and cross-domain datasets.
• Multi-layer attention (last 2–3 levels) yields notable gains over non-attention baselines and prior attention methods (e.g., GAP, PAN).
• Binarised attention maps from the proposed method outperform other CNN-based attention maps, traditional saliency maps, and top object proposals for weakly supervised segmentation on Object Discovery.
• Attention-aware models show improved robustness to adversarial perturbations at low to moderate L∞ norms, with the gap narrowing at higher perturbation levels.
• Attention maps focus on object regions and suppress background, with layer-wise specialization (lower layers for surroundings, higher layers for central object).
• Cross-domain results indicate consistent improvements (average margin around 6%) when transferring CIFAR-based models to unseen datasets.