[Paper Review] Editing a classifier by rewriting its prediction rules

Shibani Santurkar, Dimitris Tsipras+4 more|arXiv (Cornell University)|Dec 2, 2021

Data Stream Mining Techniques4 citations

TL;DR

This paper introduces a method to directly edit a classifier's prediction rules by rewriting feature representations in its hidden layers, enabling behavior modification without additional data collection. The approach corrects model failures on spurious correlations—such as misclassifying snowy scenes or adversarial typographic attacks—using only a single synthetic example, significantly improving robustness and generalization beyond fine-tuning.

ABSTRACT

We present a methodology for modifying the behavior of a classifier by directly rewriting its prediction rules. Our approach requires virtually no additional data collection and can be applied to a variety of settings, including adapting a model to new environments, and modifying it to ignore spurious features. Our code is available at https://github.com/MadryLab/EditingClassifiers .

Motivation & Objective

• To address the problem of classifiers learning unreliable, context-specific prediction rules from biased training data.
• To develop a direct, data-efficient method for modifying model behavior post-training, avoiding the need for costly data collection.
• To enable targeted correction of model failures caused by spurious correlations, such as those induced by environmental changes or adversarial perturbations.
• To provide a scalable toolkit for editing classifiers by manipulating latent representations of specific concepts.
• To demonstrate the utility of the method in real-world scenarios, including domain shift and adversarial attacks.

Proposed method

• The method modifies the classifier's weight matrix to redirect the representation of a target concept (e.g., 'snow') to match that of a reference concept (e.g., 'asphalt road') in a specific layer.
• It uses a single synthetic example to compute a rank-one update to the layer’s weights, minimizing interference with other representations.
• The editing process is constrained using a segmentation mask to preserve the original mapping in regions not containing the target concept, improving generalization.
• Concepts are identified using pre-trained instance segmentation models (e.g., on MS-COCO or LVIS), enabling automated discovery of high-level features.
• The approach leverages a concept-transformation pipeline combining style transfer and segmentation to generate diverse test cases for evaluation.
• The method is applied to both vision models (e.g., VGG16, ResNet) and zero-shot models (e.g., CLIP), demonstrating broad applicability.

Experimental results

Research questions

• RQ1 Can we modify a classifier’s prediction rules without collecting new data, by directly rewriting its internal representations?
• RQ2 How effective is the method in correcting model failures due to spurious correlations, such as those caused by environmental changes like snow?
• RQ3 To what extent can the method generalize to unseen examples and new concepts beyond the training distribution?
• RQ4 How does the method compare to standard fine-tuning in correcting model errors induced by concept-level transformations?
• RQ5 Can the same editing framework be used to probe and identify spurious prediction rules in trained models?

Key findings

• The method corrects a significant fraction of model failures induced by concept-level transformations—such as replacing 'road' with 'snow'—outperforming standard fine-tuning, which often increases errors.
• Editing with a mask during the update process leads to better performance by preserving original representations in non-target regions, suggesting a regularizing effect.
• On ImageNet and Places365 classifiers, the method reduces accuracy drops caused by concept transformations, with VGG16 showing up to 30% accuracy loss on 'three-toed sloth' when 'tree' is modified, which the method successfully mitigates.
• The approach successfully defends against the 'typographic attack' on CLIP by editing the model to ignore the text 'iPod' on physical stickers, restoring robustness.
• The concept-transformation pipeline enables scalable generation of counterfactual images, revealing model reliance on specific concepts and exposing spurious correlations.
• Models show higher sensitivity to texture-based transformations (e.g., 'graffiti', 'fall colors') than to material-based ones (e.g., 'wooden', 'metallic'), which the method can effectively correct.