[Paper Review] Fooling Neural Network Interpretations via Adversarial Model Manipulation

Juyeon Heo, Sunghwan Joo+1 more|arXiv (Cornell University)|Feb 6, 2019

Adversarial Robustness in Machine Learning35 references73 citations

TL;DR

The paper shows that state-of-the-art saliency-based interpreters (LRP, Grad-CAM, SimpleGrad) can be fooled by fine-tuning pre-trained models without hurting accuracy, via passive and active manipulation that alters explanations and transfers across methods.

ABSTRACT

We ask whether the neural network interpretation methods can be fooled via adversarial model manipulation, which is defined as a model fine-tuning step that aims to radically alter the explanations without hurting the accuracy of the original models, e.g., VGG19, ResNet50, and DenseNet121. By incorporating the interpretation results directly in the penalty term of the objective function for fine-tuning, we show that the state-of-the-art saliency map based interpreters, e.g., LRP, Grad-CAM, and SimpleGrad, can be easily fooled with our model manipulation. We propose two types of fooling, Passive and Active, and demonstrate such foolings generalize well to the entire validation set as well as transfer to other interpretation methods. Our results are validated by both visually showing the fooled explanations and reporting quantitative metrics that measure the deviations from the original explanations. We claim that the stability of neural network interpretation method with respect to our adversarial model manipulation is an important criterion to check for developing robust and reliable neural network interpretation method.

Motivation & Objective

• Assess stability of neural network interpretation methods under adversarial model manipulation.
• Demonstrate that popular saliency-based interpreters can be fooled on standard architectures (VGG19, ResNet50, DenseNet121).
• Characterize passive and active fooling schemes and their transferability across interpretation methods.
• Evaluate fooling robustness on large-scale data (ImageNet) and discuss implications for reliability of explanations.

Proposed method

• Fine-tune pre-trained models with a combined objective: classification loss plus an interpretation-based penalty term.
• Define passive fooling (Location, Top-k, Center-mass) to produce uninformative explanations.
• Define active fooling to swap explanations between two target classes using a dedicated fooling dataset.
• Use three interpreters (LRP-Composite, Grad-CAM, SimpleGrad) to generate heatmaps for explanations.
• Evaluate Fooling Success Rate (FSR) on ImageNet with held-out validation data, using predefined thresholds per fooling type.
• Assess robustness via AOPC and experiments on adversarial training.

Experimental results

Research questions

• RQ1 Can interpretation methods like LRP, Grad-CAM, and SimpleGrad reliably reflect model rationales after targeted model fine-tuning?
• RQ2 Do passive foolings (Location, Top-k, Center-mass) degrade interpretability without materially affecting accuracy?
• RQ3 Can active fooling swap explanations between classes, and is such manipulation transferable across interpreters?
• RQ4 How does fooling behavior vary across architectures (VGG19, ResNet50, DenseNet121) and with defensive training such as adversarial training?

Key findings

• Interpretation methods are vulnerable to adversarial model manipulation with minimal accuracy loss (Top-1 ~2%, Top-5 ~1%).
• Passive foolings can consistently mislead explanations across a validation set and transfer between interpreters (LRP T, Grad-CAM, SimpleG T).
• Active fooling can swap explanations between two classes, with varying success across architectures (high success for VGG19/ResNet50, limited for DenseNet121).
• Fooling transfers across interpreters and to other architectures, indicating a systemic stability issue in saliency-based explanations.
• The manipulation can be difficult to detect or undo by simple perturbations or adversarial training, and can persist under Gaussian perturbations.