[Paper Review] Zoom-CAM: Generating Fine-grained Pixel Annotations from Image Labels

Xiangwei Shi, Seyran Khademi+2 more|arXiv (Cornell University)|Oct 16, 2020

Advanced Neural Network Applications37 references4 citations

TL;DR

Zoom-CAM improves weakly supervised object localization and segmentation by integrating class activation maps from all intermediate convolutional layers, not just the final one, to generate higher-resolution, fine-grained pixel-level pseudo-labels. This approach reduces localization noise and captures small-scale objects missed by baseline methods, achieving a 2.8% top-1 error reduction on ImageNet and a 1.1% mIoU improvement in weakly supervised semantic segmentation.

ABSTRACT

Current weakly supervised object localization and segmentation rely on class-discriminative visualization techniques to generate pseudo-labels for pixel-level training. Such visualization methods, including class activation mapping (CAM) and Grad-CAM, use only the deepest, lowest resolution convolutional layer, missing all information in intermediate layers. We propose Zoom-CAM: going beyond the last lowest resolution layer by integrating the importance maps over all activations in intermediate layers. Zoom-CAM captures fine-grained small-scale objects for various discriminative class instances, which are commonly missed by the baseline visualization methods. We focus on generating pixel-level pseudo-labels from class labels. The quality of our pseudo-labels evaluated on the ImageNet localization task exhibits more than 2.8% improvement on top-1 error. For weakly supervised semantic segmentation our generated pseudo-labels improve a state of the art model by 1.1%.

Motivation & Objective

• To address the limitation of existing weakly supervised methods that rely only on the final, low-resolution convolutional layer for pseudo-label generation.
• To improve localization accuracy for small-scale and fine-grained object instances commonly missed by Grad-CAM and CAM.
• To develop a method that leverages feature maps from all intermediate layers to generate high-resolution, precise visual explanations from image-level labels.
• To demonstrate that integrating multiple layers reduces noise and enhances localization fidelity in weakly supervised semantic segmentation.

Proposed method

• Zoom-CAM computes gradient-based importance maps by back-propagating the class score gradient through all intermediate convolutional layers, not just the last one.
• It applies learnable weight masks to linearly combine activation maps across all intermediate layers, optimizing for class-specific saliency.
• The method resizes the fused activation maps to input image resolution to generate high-resolution visual explanations for pseudo-labeling.
• It uses a thresholding and connected component analysis strategy to extract salient regions from the fused maps for use as pseudo-segmentation labels.
• The approach is compatible with standard CNNs and adds minimal computational overhead beyond standard back-propagation.
• It replaces CAM or Grad-CAM in weakly supervised pipelines by generating more accurate pseudo-labels from image-level supervision.

Experimental results

Research questions

• RQ1 Can integrating feature maps from all intermediate convolutional layers improve the precision of pseudo-labels in weakly supervised localization?
• RQ2 Does using multiple layers reduce localization noise and better capture small-scale or fine-grained object instances compared to relying on only the final layer?
• RQ3 How does the performance of Zoom-CAM compare to state-of-the-art visualization methods like Grad-CAM and Score-CAM in terms of localization and segmentation accuracy?
• RQ4 Can Zoom-CAM-generated pseudo-labels improve the performance of a state-of-the-art weakly supervised semantic segmentation model?

Key findings

• Zoom-CAM reduces top-1 localization error by 2.8% and top-5 error by 3.7% on the ImageNet localization benchmark compared to Grad-CAM.
• On the PASCAL VOC 2012 dataset, Zoom-CAM-generated pseudo-labels achieve a higher mIoU than those from CAM, Grad-CAM, Score-CAM, and Grad-CAM++.
• When used to re-train the state-of-the-art IRNet model for weakly supervised semantic segmentation, Zoom-CAM improves mIoU by 1.1%.
• Visual inspection confirms that Zoom-CAM successfully localizes multiple instances of the same class and small-scale objects that are missed by Grad-CAM.
• The method maintains computational efficiency similar to Grad-CAM, as the main cost remains back-propagation.
• The integration of intermediate layer maps reduces background over-activation and improves spatial localization fidelity.