[Paper Review] Explaining Convolutional Neural Networks using Softmax Gradient Layer-wise Relevance Propagation

Brian Kenji Iwana, Ryohei Kuroki+1 more|arXiv (Cornell University)|Aug 6, 2019

Explainable Artificial Intelligence (XAI)42 references21 citations

TL;DR

This paper proposes Softmax-Gradient Layer-wise Relevance Propagation (SGLRP), a novel method that enhances Layer-wise Relevance Propagation (LRP) by using the gradient of the softmax output to backpropagate relevance, enabling class-discriminative pixel-wise attribution. SGLRP outperforms existing LRP-based methods in localizing target objects and suppressing irrelevant features, particularly in multi-class scenarios with occlusions or competing classes.

ABSTRACT

Convolutional Neural Networks (CNN) have become state-of-the-art in the field of image classification. However, not everything is understood about their inner representations. This paper tackles the interpretability and explainability of the predictions of CNNs for multi-class classification problems. Specifically, we propose a novel visualization method of pixel-wise input attribution called Softmax-Gradient Layer-wise Relevance Propagation (SGLRP). The proposed model is a class discriminate extension to Deep Taylor Decomposition (DTD) using the gradient of softmax to back propagate the relevance of the output probability to the input image. Through qualitative and quantitative analysis, we demonstrate that SGLRP can successfully localize and attribute the regions on input images which contribute to a target object's classification. We show that the proposed method excels at discriminating the target objects class from the other possible objects in the images. We confirm that SGLRP performs better than existing Layer-wise Relevance Propagation (LRP) based methods and can help in the understanding of the decision process of CNNs.

Motivation & Objective

• To address the interpretability challenge of deep CNNs, particularly their 'black box' nature in multi-class image classification.
• To improve existing LRP-based visualization methods by introducing class discrimination to suppress irrelevant or conflicting features.
• To develop a method that accurately attributes input pixel relevance to specific target classes, even when multiple classes are present.
• To evaluate the method's performance in both qualitative and quantitative settings, including localization and robustness to occlusions.

Proposed method

• SGLRP extends LRP by using the gradient of the softmax output as the initial relevance signal, enabling class-specific attribution.
• The method backpropagates relevance through the network using the gradient of the softmax function, which inherently accounts for non-target class probabilities.
• Relevance is redistributed layer-by-layer using a modified version of Deep Taylor Decomposition (DTD), with the gradient of softmax as the starting point.
• The approach subtracts relevance from non-target classes by leveraging the derivative of the softmax function, enhancing class discrimination.
• The method produces pixel-wise relevance maps that highlight regions contributing to the predicted class while suppressing background and distractor objects.
• SGLRP is implemented using the iNNvestigate library and applied to VGG16 for evaluation on ImageNet.

Experimental results

Research questions

• RQ1 Can SGLRP produce more class-discriminative relevance maps than existing LRP-based methods in multi-class image classification?
• RQ2 How does SGLRP perform in localizing the target object when multiple classes are present or when the target is occluded?
• RQ3 Does using the gradient of softmax as the initial relevance signal improve robustness and accuracy in attribution compared to standard LRP?
• RQ4 How does SGLRP compare to other explainability methods like Guided Grad-CAM and CLRP in terms of localization accuracy and energy distribution?

Key findings

• SGLRP achieved the highest pointing accuracy among all LRP-based methods at energy levels ≥67%, indicating superior localization precision.
• At energy levels below 67%, Guided Grad-CAM outperformed SGLRP, but SGLRP surpassed it at higher energy thresholds, showing better robustness to noise and background artifacts.
• SGLRP outperformed CLRP and other LRP variants in the patch removal evaluation, especially when the second-highest probability class was targeted, demonstrating strong class discrimination.
• The method successfully localized target objects even in complex scenes with multiple distractors, such as the spider in (a), person in (b), and guitars in (c) of Figure 7.
• Random baseline showed near-zero pointing accuracy, confirming that SGLRP’s performance is not due to chance.
• SGLRP’s use of softmax gradient enabled effective suppression of non-target class relevance, reducing false positives in attribution maps.