[Paper Review] Extracting Class Activation Maps from Non-Discriminative Features as well

Zhaozheng Chen, Qianru Sun|arXiv (Cornell University)|Mar 18, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Local Prototype CAM (LPCAM), a novel method to generate class activation maps (CAMs) that capture both discriminative and non-discriminative local features by clustering local features into prototypes. By replacing the standard classifier weight with prototype-based similarity maps and applying independent normalization per prototype, LPCAM achieves complete object coverage in weakly-supervised semantic segmentation, outperforming state-of-the-art methods on PASCAL VOC and MS COCO with minimal computational overhead.

ABSTRACT

Extracting class activation maps (CAM) from a classification model often results in poor coverage on foreground objects, i.e., only the discriminative region (e.g., the "head" of "sheep") is recognized and the rest (e.g., the "leg" of "sheep") mistakenly as background. The crux behind is that the weight of the classifier (used to compute CAM) captures only the discriminative features of objects. We tackle this by introducing a new computation method for CAM that explicitly captures non-discriminative features as well, thereby expanding CAM to cover whole objects. Specifically, we omit the last pooling layer of the classification model, and perform clustering on all local features of an object class, where "local" means "at a spatial pixel position". We call the resultant K cluster centers local prototypes - represent local semantics like the "head", "leg", and "body" of "sheep". Given a new image of the class, we compare its unpooled features to every prototype, derive K similarity matrices, and then aggregate them into a heatmap (i.e., our CAM). Our CAM thus captures all local features of the class without discrimination. We evaluate it in the challenging tasks of weakly-supervised semantic segmentation (WSSS), and plug it in multiple state-of-the-art WSSS methods, such as MCTformer and AMN, by simply replacing their original CAM with ours. Our extensive experiments on standard WSSS benchmarks (PASCAL VOC and MS COCO) show the superiority of our method: consistent improvements with little computational overhead.

Motivation & Objective

• To address the poor foreground object coverage in conventional class activation maps (CAMs), which often miss non-discriminative regions like legs or tails of objects.
• To overcome the bias of standard CAMs toward only discriminative features by replacing the classifier weight with a prototype-based representation.
• To develop a method that preserves both local semantics (e.g., 'head', 'leg', 'body') and mitigates false positives from confusing contexts (e.g., 'rail' for 'train').
• To create a plug-and-play alternative to conventional CAMs that enhances performance in weakly-supervised semantic segmentation (WSSS) with minimal computational cost.

Proposed method

• Omit the global average pooling layer in the classification model to retain all spatially localized features before the final classifier.
• Cluster all local features (across all training images of a class) into K local prototypes, each representing a distinct local semantic part (e.g., 'head', 'leg' of 'sheep').
• For a given image, compute K similarity maps between its feature map and each local prototype using cosine distance, preserving non-discriminative regions through independent normalization per map.
• Aggregate the normalized similarity maps via averaging to produce the final heatmap, termed Local Prototype CAM (LPCAM).
• Extend LPCAM by subtracting context similarity maps (computed between image features and clustered context prototypes) to suppress false positives from highly co-occurring background elements.
• Integrate LPCAM into existing WSSS pipelines by replacing the original CAM computation step, requiring no architectural changes to the segmentation model.

Figure 1 : The CAM of image $\bm{x}$ is computed by $f(\bm{x})\cdot\mathbf{w}_{c}$ , where $f(\bm{x})$ is the feature map block (before the last pooling layer of the multi-label classification model) and $\mathbf{w}_{c}$ denotes the classifier weights of class $c$ . (a) Input images. (b) CAMs genera

Experimental results

Research questions

• RQ1 Can a prototype-based classifier derived from un-pooled features improve object coverage in CAMs compared to standard classifier weights?
• RQ2 Does the use of independent normalization per prototype-based similarity map preserve non-discriminative features that are typically ignored by conventional CAMs?
• RQ3 To what extent can LPCAM reduce false positives from confusing contexts (e.g., 'rail' for 'train') while maintaining full object coverage?
• RQ4 How does LPCAM perform when plugged into diverse state-of-the-art WSSS methods across different backbones and pre-training strategies?

Key findings

• LPCAM achieves state-of-the-art performance on the PASCAL VOC 2012 validation set with 70.1% mIoU and 70.4% on the test set when used with AMN.
• On the more challenging MS COCO dataset, AMN+LPCAM outperforms the original AMN and all prior SOTA methods based on WRN-38, achieving 45.5% mIoU.
• The method improves mIoU by 0.6–0.8 percentage points across multiple WSSS methods, including IRN, MCTformer, and EDAM, with consistent gains on both VOC and COCO.
• Sensitivity analysis shows LPCAM is robust to hyperparameter changes, with minimal performance drop across varying thresholds for feature selection and clustering.
• Qualitative results demonstrate that LPCAM captures non-discriminative parts (e.g., legs, tails) and suppresses context artifacts (e.g., 'waves' in 'surfboard' images), while occasionally over-activating strongly co-occurring contexts like 'railroad' with 'train'.

Figure 2 : The pipeline for generating LPCAM. (a) Generating local prototypes from all training images of an individual class, e.g., “sheep”. (b) Generating LPCAM by sliding both class prototypes and context prototypes over the feature block of the input image, and aggregating the obtained similarit