[Paper Review] Fighting Fire with Fire: Avoiding DNN Shortcuts through Priming

Chuan Wen, Jianing Qian+4 more|arXiv (Cornell University)|Jun 22, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes a simple yet effective method called 'priming' to help deep neural networks (DNNs) avoid learning spurious shortcuts—such as classifying backgrounds instead of objects—by injecting domain-knowledge-based auxiliary signals. By training a DNN with a priming variable derived from key input features (e.g., salient foregrounds or recent frames), the model is guided toward learning more robust, task-relevant representations, achieving state-of-the-art performance on image classification, imitation learning, and autonomous driving benchmarks.

ABSTRACT

Across applications spanning supervised classification and sequential control, deep learning has been reported to find "shortcut" solutions that fail catastrophically under minor changes in the data distribution. In this paper, we show empirically that DNNs can be coaxed to avoid poor shortcuts by providing an additional "priming" feature computed from key input features, usually a coarse output estimate. Priming relies on approximate domain knowledge of these task-relevant key input features, which is often easy to obtain in practical settings. For example, one might prioritize recent frames over past frames in a video input for visual imitation learning, or salient foreground over background pixels for image classification. On NICO image classification, MuJoCo continuous control, and CARLA autonomous driving, our priming strategy works significantly better than several popular state-of-the-art approaches for feature selection and data augmentation. We connect these empirical findings to recent theoretical results on DNN optimization, and argue theoretically that priming distracts the optimizer away from poor shortcuts by creating better, simpler shortcuts.

Motivation & Objective

• To address the pervasive problem of DNNs learning spurious shortcuts that fail under distributional shift.
• To investigate whether incorporating domain-knowledge-based priming signals can guide DNN optimization toward more robust, generalizable solutions.
• To demonstrate that priming, based on approximate key input features, can outperform existing state-of-the-art methods in feature selection and data augmentation.
• To provide theoretical justification for why priming helps avoid poor local minima by creating simpler, better-structured optimization paths.

Proposed method

• Introduce a 'priming' signal derived from key input features—such as salient foregrounds in images or recent observations in sequential tasks—using domain knowledge.
• Train a DNN end-to-end with both the full input and the priming variable, which acts as a supervision signal to bias attention toward relevant input regions.
• Use a coarse estimate of the target label (e.g., from a small backbone or heuristic) to generate the priming variable from key inputs.
• Apply the method in three settings: NICO image classification, MuJoCo continuous control, and CARLA autonomous driving.
• The priming signal is not required to be perfect; it only needs to be likely to contain task-relevant information in many cases.
• Theoretical analysis links priming to NTK and linearization theory, showing it creates a simpler, better-optimized path in the loss landscape.

Experimental results

Research questions

• RQ1 Can injecting a simple priming signal based on key input features improve DNN generalization and reduce shortcut learning?
• RQ2 How does priming compare to state-of-the-art data augmentation and feature selection techniques in avoiding distributional shift failures?
• RQ3 What is the theoretical mechanism by which priming helps the optimizer avoid poor local minima?
• RQ4 Does the method remain effective even when the priming signal is imperfect or occasionally incorrect?
• RQ5 Can priming be applied effectively across diverse tasks such as image classification, imitation learning, and autonomous driving?

Key findings

• The proposed priming method significantly outperforms state-of-the-art data augmentation and feature selection techniques on NICO image classification, MuJoCo continuous control, and CARLA autonomous driving benchmarks.
• On the NICO dataset, priming reduces shortcut learning—where models classify based on background rather than object—by guiding attention to the salient foreground.
• In CARLA autonomous driving, priming prevents 'copycat' behavior where models copy previous actions instead of reacting to new traffic signals.
• Empirical results show that priming leads to more robust generalization under distributional shift, even with coarse priming signals.
• Theoretical analysis suggests that priming creates a simpler, better-structured optimization path in the loss landscape, reducing the likelihood of converging to poor shortcuts.
• The method is simple to implement, end-to-end trainable, and effective across diverse, real-world applications without architectural changes.