[Paper Review] What do Deep Networks Like to Read?

Jonas Pfeiffer, Aishwarya Kamath+2 more|arXiv (Cornell University)|Sep 10, 2019

Topic Modeling36 references4 citations

TL;DR

This paper introduces SIFT (Susceptibility Identification through Fine-tuning), a novel abstractive method that uncovers deep neural network preferences in NLP by fine-tuning an autoencoder with gradients from a fixed classifier. It reveals model-specific propensities and artifacts—such as bias-inducing terms—without prior assumptions, demonstrating strong correlation with known data artifacts and uncovering new ones across multiple models and tasks.

ABSTRACT

Recent research towards understanding neural networks probes models in a top-down manner, but is only able to identify model tendencies that are known a priori. We propose Susceptibility Identification through Fine-Tuning (SIFT), a novel abstractive method that uncovers a model's preferences without imposing any prior. By fine-tuning an autoencoder with the gradients from a fixed classifier, we are able to extract propensities that characterize different kinds of classifiers in a bottom-up manner. We further leverage the SIFT architecture to rephrase sentences in order to predict the opposing class of the ground truth label, uncovering potential artifacts encoded in the fixed classification model. We evaluate our method on three diverse tasks with four different models. We contrast the propensities of the models as well as reproduce artifacts reported in the literature.

Motivation & Objective

• To address the gap in understanding neural network preferences beyond extractive rationales or predefined probing tasks.
• To develop a bottom-up, abstractive method that uncovers what classifiers inherently 'like to read' without imposing prior knowledge.
• To identify and analyze model-specific biases and artifacts encoded in pre-trained classifiers using a unified framework.
• To compare model preferences across different architectures (LSTM, CNN, DAN) and tasks (sentiment, NLI, PubMed classification).
• To evaluate the method’s ability to generate counterfactual inputs that reverse classifier predictions by targeting encoded artifacts.

Proposed method

• Fine-tune an autoencoder (AE) using gradients from a fixed, pretrained classifier via the Straight-Through Gumbel-Softmax estimator.
• Use the AE to reconstruct input sequences in a way that preserves only information deemed useful by the classifier, revealing its preferences.
• Apply the method to both sentence-level and sentence-pair classification tasks (e.g., SST-2, SNLI, PubMed) to extract model propensities.
• Introduce a reconstruction loss that penalizes large cosine distance between original and generated sentence embeddings to ensure coherence.
• Flip ground-truth labels during fine-tuning to generate inputs that cause the classifier to predict the opposite class, exposing encoded artifacts.
• Compare SIFT-extracted terms with those from PMI (pointwise mutual information) to assess alignment with data-level artifacts and model-level encoding.

Experimental results

Research questions

• RQ1 What kinds of linguistic patterns or terms do different neural network architectures prefer when making predictions?
• RQ2 To what extent do classifiers encode data-level artifacts (e.g., specific words or phrases) that influence predictions?
• RQ3 Can an abstractive, unsupervised method like SIFT uncover model-specific biases and preferences without prior assumptions?
• RQ4 How well do SIFT-generated counterfactual inputs align with known artifacts reported in prior work?
• RQ5 Does SIFT reveal novel artifacts beyond those identified by traditional statistical methods like PMI?

Key findings

• SIFT successfully identifies model-specific propensities correlated with architecture, such as LSTM models favoring sentiment-laden terms like 'love' or 'bad'.
• In the SST-2 sentiment task, SIFT terms showed a weighted Kendall’s tau correlation of 0.486 with PMI terms, indicating alignment with known data-level artifacts.
• For the PubMed classification task, SIFT uncovered contextually relevant terms (e.g., 'objective' → 'conclusion') that did not correlate with PMI, suggesting encoding of deeper semantic patterns from pretraining.
• On the SNLI task, SIFT identified known artifacts like 'sleeping', 'cats', and 'cat' as influential, with a correlation of 0.366 with PMI, confirming its sensitivity to known biases.
• The method achieved 98–99% accuracy in generating inputs that reverse the classifier’s prediction in the flipped-label setting, demonstrating its effectiveness in eliciting model behavior.
• SIFT revealed that models encode not just surface-level keywords but also semantic and contextual cues, especially when pre-trained on large-scale unlabeled data.