[Paper Review] Learning Deep Features for Discriminative Localization

Bolei Zhou, Aditya Khosla+3 more|arXiv (Cornell University)|Dec 14, 2015

Advanced Neural Network ApplicationsComputer Science28 references206 citations

TL;DR

This paper introduces Class Activation Mapping (CAM) to enable convolutional neural networks (CNNs) trained on image-level labels to perform weakly supervised object localization through global average pooling (GAP). By leveraging GAP and class-specific softmax weights, CAM generates class-discriminative localization maps in a single forward pass, achieving 37.1% top-5 error on ILSVRC 2014—remarkably close to fully supervised methods—demonstrating that deep features can be both interpretable and transferable across tasks without bounding box annotations.

ABSTRACT

In this work, we revisit the global average pooling layer proposed in [13], and shed light on how it explicitly enables the convolutional neural network to have remarkable localization ability despite being trained on image-level labels. While this technique was previously proposed as a means for regularizing training, we find that it actually builds a generic localizable deep representation that can be applied to a variety of tasks. Despite the apparent simplicity of global average pooling, we are able to achieve 37.1% top-5 error for object localization on ILSVRC 2014, which is remarkably close to the 34.2% top-5 error achieved by a fully supervised CNN approach. We demonstrate that our network is able to localize the discriminative image regions on a variety of tasks despite not being trained for them

Motivation & Objective

• To enable CNNs trained on image-level labels to localize discriminative image regions without bounding box annotations.
• To investigate whether global average pooling (GAP) can preserve and enhance localization ability in deep networks beyond regularization.
• To demonstrate that the resulting deep features are generic and transferable across diverse visual recognition tasks.
• To provide an interpretable visualization of class-specific features using class activation maps (CAM) for better understanding of CNN decision-making.

Proposed method

• Replace fully connected layers with global average pooling (GAP) to maintain spatial feature maps and enable localization.
• Use the final convolutional layer’s feature maps and the corresponding class-specific softmax weights to generate class activation maps (CAM).
• Apply a weighted sum of feature maps using the learned classifier weights to produce a class-discriminative heatmap.
• Train the network end-to-end using standard image classification loss, enabling localization in a single forward pass.
• Utilize the CAM visualization to interpret which image regions are most relevant for classification.
• Transfer the learned localizable features to zero-shot tasks such as action recognition, scene classification, and concept discovery.

Experimental results

Research questions

• RQ1 Can a CNN trained only on image-level labels localize discriminative object regions without any bounding box supervision?
• RQ2 How does global average pooling (GAP) contribute to preserving localization capability compared to global max pooling or fully connected layers?
• RQ3 To what extent can the localization ability of a classification-trained CNN generalize to tasks it was not explicitly trained for?
• RQ4 Can class activation maps (CAM) effectively visualize and interpret the discriminative regions used by the network for classification?
• RQ5 How does the use of average pooling improve localization accuracy compared to max pooling in weakly supervised settings?

Key findings

• The proposed CAM method achieves 37.1% top-5 error on the ILSVRC 2014 weakly supervised object localization benchmark, which is very close to the 34.2% top-5 error of a fully supervised AlexNet.
• Global average pooling enables the network to retain localization capability through the final layers, unlike fully connected layers which destroy spatial information.
• The method localizes discriminative regions for tasks such as action recognition and text detection even though the network was not trained on those tasks.
• Class activation maps successfully highlight semantically meaningful regions—such as the toothbrush in a brushing action or text in street view images—without any bounding box annotations.
• The approach enables visualization of class-specific units in deep CNNs, revealing which feature maps are most discriminative for specific classes, such as dog face or sofa in scene recognition.
• The technique generalizes to zero-shot applications, including visual question answering, where CAM highlights image regions relevant to the predicted answer.