[Paper Review] Tell Me Where to Look: Guided Attention Inference Network

Kunpeng Li, Ziyan Wu+3 more|arXiv (Cornell University)|Feb 27, 2018

Advanced Neural Network ApplicationsComputer Science36 references76 citations

TL;DR

The paper introduces GAIN, an end-to-end framework that makes attention maps trainable and guides them with self-supervision and optional extra supervision to improve weakly supervised semantic segmentation, achieving state-of-the-art results on VOC 2012.

ABSTRACT

Weakly supervised learning with only coarse labels can obtain visual explanations of deep neural network such as attention maps by back-propagating gradients. These attention maps are then available as priors for tasks such as object localization and semantic segmentation. In one common framework we address three shortcomings of previous approaches in modeling such attention maps: We (1) first time make attention maps an explicit and natural component of the end-to-end training, (2) provide self-guidance directly on these maps by exploring supervision form the network itself to improve them, and (3) seamlessly bridge the gap between using weak and extra supervision if available. Despite its simplicity, experiments on the semantic segmentation task demonstrate the effectiveness of our methods. We clearly surpass the state-of-the-art on Pascal VOC 2012 val. and test set. Besides, the proposed framework provides a way not only explaining the focus of the learner but also feeding back with direct guidance towards specific tasks. Under mild assumptions our method can also be understood as a plug-in to existing weakly supervised learners to improve their generalization performance.

Motivation & Objective

• Motivate learning with only image-level labels and obtain reliable attention maps for localization and segmentation.
• Make attention maps an explicit, trainable component during end-to-end training.
• Provide self-guidance to expand attention beyond the most discriminative regions.
• Enable integration of extra supervision to bridge weak and full supervision.
• Demonstrate state-of-the-art performance on PASCAL VOC 2012 segmentation under weak supervision.

Proposed method

• Two-stream network with shared parameters: a classification stream (S_cl) and an attention mining stream (S_am).
• Attention maps are generated online via Grad-CAM-like mechanisms using class-score gradients and global average pooled weights to form A^c.
• A soft mask I*^c is derived from A^c to constrain S_am and encourage exploration beyond the most discriminative regions (Attention Mining Loss, L_am).
• Self-guidance loss L_self = L_cl + α L_am enforces attention to cover more of the object; α is a weighting parameter (α = 1 used).
• GAIN ext extends the framework by incorporating external supervision L_e (e.g., pixel-level masks) to further tailor attention maps, yielding L_ext = L_cl + α L_am + ω L_e (ω = 10 in experiments).
• During training, attention maps serve as priors for weakly supervised segmentation frameworks (e.g., SEC), enabling improved localization cues without full supervision.

Experimental results

Research questions

• RQ1 Can attention maps be made an explicit, trainable component during end-to-end training for weakly supervised tasks?
• RQ2 Does self-guidance on attention maps promote more complete object coverage beyond the most discriminative regions?
• RQ3 Can integrating extra supervision on attention maps further improve performance and robustness to bias in training data?
• RQ4 What impact does guided attention have on segmentation performance under weak supervision on VOC 2012?
• RQ5 Is the GAIN framework compatible as a plug-in to existing weakly supervised learners to boost generalization?

Key findings

• GAIN achieves state-of-the-art mIoU on VOC 2012 val and test under weak supervision (55.3% val, 56.8% test for GAIN).
• GAIN ext with small pixel-level supervision further improves mIoU to 60.5% (val) and 62.1% (test).
• Without pixel-level labels, GAIN-based SEC outperforms several weakly supervised methods, demonstrating the benefit of trainable attention maps.
• Adding pixel-level supervision in GAIN ext yields performance gains up to 4.6–4.1 percentage points over competing methods under similar settings.
• Qualitative results show GAIN expands attention to more complete object regions, improving segmentation priors.