[Paper Review] SS-CAM: Smoothed Score-CAM for Sharper Visual Feature Localization

Haofan Wang, Rakshit Naidu+2 more|arXiv (Cornell University)|Jun 25, 2020

Explainable Artificial Intelligence (XAI)Computer Science28 references52 citations

TL;DR

SS-CAM enhances Score-CAM by smoothing activation maps to produce sharper, more centralized localization of object features, improving faithfulness and localization on ImageNet.

ABSTRACT

Interpretation of the underlying mechanisms of Deep Convolutional Neural Networks has become an important aspect of research in the field of deep learning due to their applications in high-risk environments. To explain these black-box architectures there have been many methods applied so the internal decisions can be analyzed and understood. In this paper, built on the top of Score-CAM, we introduce an enhanced visual explanation in terms of visual sharpness called SS-CAM, which produces centralized localization of object features within an image through a smooth operation. We evaluate our method on the ILSVRC 2012 Validation dataset, which outperforms Score-CAM on both faithfulness and localization tasks.

Motivation & Objective

• Motivate interpretable explanations for CNN decisions in high-stakes settings.
• Develop a sharper, centralized localization method built on Score-CAM by applying smoothing.
• Evaluate the method on ILSVRC 2012 to assess faithfulness, localization, and human trust.
• Compare SS-CAM against Grad-CAM, Grad-CAM++, Smooth Grad-CAM++, and Score-CAM on multiple metrics.

Proposed method

• Extend Score-CAM by incorporating smoothing over noisy samples to obtain stable activation weights.
• Introduce two smoothing strategies: smoothing on feature space (type1) and smoothing on input space (type2).
• Define Channel-wise Increase of Confidence (CIC) to quantify activation map importance.
• Compute final attribution as a ReLU-weighted sum of activation maps with smoothed CIC-derived weights.
• Normalize and upsample activation maps, then average scores over N noisy samples to obtain alpha_k.
• Provide two equations for alpha_k: (type1) alpha_k = (1/N) sum C(M) over noisy samples, with M = sum_N (X0 * (A_l^k + N(0,σ))); (type2) alpha_k = (1/N) sum C(M) where M = sum_N ((X0 * A_l^k) + N(0,σ)).
• Normalization: s(A_l^k) = (A_l^k - min(A_l^k)) / (max(A_l^k) - min(A_l^k)).

Experimental results

Research questions

• RQ1 Does SS-CAM produce sharper and more localized attribution maps than Score-CAM and other CAM-based methods?
• RQ2 Do the smoothing strategies improve faithfulness, localization, and human interpretability without excessive Computational cost?
• RQ3 How do SS-CAM variants fare across standard metrics like deletion/insertion curves and energy-based pointing game?

Key findings

• SS-CAM variants achieve improved localization and sharper attribution maps compared to Score-CAM, as shown by qualitative visualizations.
• In faithfulness tests on 2000 ILSVRC-2012 images with VGG-16, SS-CAM variants show competitive Average Drop and higher or comparable Average Increase in Confidence compared to Grad-CAM/Grad-CAM++.
• Energy-based pointing game results indicate SS-CAM provides better or comparable localization across VGG-16 and ResNet-18, with some improvements over Score-CAM.
• Human trust evaluations favor SS-CAM(2) over other methods, indicating higher interpretability of the explanations.
• Insertion curves for SS-CAM demonstrate a notable score increase as pixels are inserted, indicating strong explanatory usefulness.
• SS-CAM(2) generally achieves a favorable balance between fidelity and sharp localization, though SS-CAM(1) sometimes surpasses Score-CAM on certain metrics.