[Paper Review] Gradient-based Analysis of NLP Models is Manipulable

Junlin Wang, Jens Tuyls+2 more|arXiv (Cornell University)|Oct 12, 2020

Adversarial Robustness in Machine Learning38 references4 citations

TL;DR

This paper demonstrates that gradient-based interpretability methods in NLP are highly manipulable by introducing a 'Facade' model with misleading gradients that can be merged with any target model. Despite preserving the original model's predictions, the merged model's gradients dominate in favor of irrelevant tokens (e.g., stop words or first tokens), rendering saliency maps, input reduction, and adversarial attacks ineffective and deceptive—undermining the faithfulness of gradient-based analysis in adversarial settings.

ABSTRACT

Gradient-based analysis methods, such as saliency map visualizations and adversarial input perturbations, have found widespread use in interpreting neural NLP models due to their simplicity, flexibility, and most importantly, their faithfulness. In this paper, however, we demonstrate that the gradients of a model are easily manipulable, and thus bring into question the reliability of gradient-based analyses. In particular, we merge the layers of a target model with a Facade that overwhelms the gradients without affecting the predictions. This Facade can be trained to have gradients that are misleading and irrelevant to the task, such as focusing only on the stop words in the input. On a variety of NLP tasks (text classification, NLI, and QA), we show that our method can manipulate numerous gradient-based analysis techniques: saliency maps, input reduction, and adversarial perturbations all identify unimportant or targeted tokens as being highly important. The code and a tutorial of this paper is available at http://ucinlp.github.io/facade.

Motivation & Objective

• To investigate the robustness of gradient-based interpretation methods in NLP under adversarial manipulation.
• To demonstrate that model gradients can be manipulated to mislead interpretation techniques without altering model predictions.
• To expose the fragility of widely used gradient-based analysis tools such as saliency maps, input reduction, and adversarial perturbations.
• To highlight the risk that model developers could conceal model biases using gradient manipulation.
• To encourage the development of more robust and trustworthy interpretation techniques.

Proposed method

• A 'Facade' model is trained to produce strong, misleading gradients (e.g., focusing on stop words or the first token) while generating low, uniform predictions for the task.
• The Facade model is merged with a target model by combining their layers, preserving the original model’s logits and predictions.
• The merged model inherits the target model’s prediction behavior but is dominated by the Facade’s gradient patterns.
• The method is applied across multiple NLP tasks: text classification, natural language inference (NLI), and question answering (QA).
• Gradient-based analysis techniques—including saliency maps, input reduction, and HotFlip adversarial attacks—are evaluated on the merged model to assess manipulation effects.
• The approach assumes white-box access, where the adversary controls the model’s internal structure and can insert the Facade during training or fine-tuning.

Experimental results

Research questions

• RQ1 Can gradient-based interpretation methods be manipulated to misattribute importance to irrelevant input tokens without changing model predictions?
• RQ2 To what extent do saliency maps, input reduction, and adversarial attacks remain reliable when gradients are dominated by a misleading Facade model?
• RQ3 Can a Facade model be constructed to make gradients focus on stop words or the first token while preserving the original model’s output?
• RQ4 How does gradient manipulation affect the faithfulness of widely used interpretability tools in NLP?
• RQ5 Can such manipulation conceal model biases, such as gender bias in profession classification, from gradient-based audits?

Key findings

• The merged model maintains nearly identical predictions to the original model, confirming that the target behavior is preserved despite gradient manipulation.
• Saliency maps on the merged model incorrectly highlight stop words or the first token as most important, even when they are irrelevant to the prediction.
• Input reduction techniques on the merged model consistently identify and remove unimportant tokens, such as stop words, leading to nonsensical reduced inputs.
• HotFlip adversarial attacks on the merged model predominantly perturb stop words or the first token, failing to target meaningful input features.
• The manipulation successfully conceals gender bias in a profession classification task, where the model’s biased behavior is hidden from gradient-based analysis.
• InteGrad, a gradient-based method that averages gradients over multiple inputs, is found to be the most robust to this manipulation among the evaluated gradient techniques.