[Paper Review] Informative Dropout for Robust Representation Learning: A Shape-bias Perspective

Baifeng Shi, Dinghuai Zhang+4 more|arXiv (Cornell University)|Aug 10, 2020

Adversarial Robustness in Machine Learning46 citations

TL;DR

The paper introduces Informative Dropout (InfoDrop), a lightweight, model-agnostic method that reduces CNN texture-bias by dropping low-information input regions, yielding improved robustness across domain generalization, few-shot classification, image corruption, and adversarial scenarios.

ABSTRACT

Convolutional Neural Networks (CNNs) are known to rely more on local texture rather than global shape when making decisions. Recent work also indicates a close relationship between CNN's texture-bias and its robustness against distribution shift, adversarial perturbation, random corruption, etc. In this work, we attempt at improving various kinds of robustness universally by alleviating CNN's texture bias. With inspiration from the human visual system, we propose a light-weight model-agnostic method, namely Informative Dropout (InfoDrop), to improve interpretability and reduce texture bias. Specifically, we discriminate texture from shape based on local self-information in an image, and adopt a Dropout-like algorithm to decorrelate the model output from the local texture. Through extensive experiments, we observe enhanced robustness under various scenarios (domain generalization, few-shot classification, image corruption, and adversarial perturbation). To the best of our knowledge, this work is one of the earliest attempts to improve different kinds of robustness in a unified model, shedding new light on the relationship between shape-bias and robustness, also on new approaches to trustworthy machine learning algorithms. Code is available at https://github.com/bfshi/InfoDrop.

Motivation & Objective

• Motivate and quantify the link between CNN texture-bias and non-robustness across distribution shifts, corruptions, and adversarial perturbations.
• Propose a light-weight, plug-in mechanism (InfoDrop) to reduce texture-bias and enhance shape-bias.
• Demonstrate universal robustness improvements when InfoDrop is applied, and its compatibility with other robustness techniques.

Proposed method

• Define self-information of local patches and approximate its distribution via kernel density estimation over a local neighborhood.
• Compute a drop probability for each output activation using a Boltzmann form r(z) ∝ exp(-I(p)/T), where I is self-information and T is a temperature parameter.
• Apply a dropout-like process at training time that zeroes high-information regions less often and low-information regions more often, effectively decorrelating outputs from texture-like inputs.
• Maintain a standard convolutional forward pass while stochastically masking outputs based on patch information content.
• During inference, remove InfoDrop and optionally finetune on clean data to assess emergent shape-bias without reliance on InfoDrop at test time.

Experimental results

Research questions

• RQ1 Does reducing local texture bias via InfoDrop induce a more shape-biased representation in CNNs?
• RQ2 Can InfoDrop improve robustness to domain shifts, few-shot generalization, random corruptions, and adversarial perturbations?
• RQ3 Is InfoDrop compatible with existing robustness techniques and beneficial across diverse architectures and tasks?

Key findings

• InfoDrop yields consistent improvements over baselines across domain generalization, few-shot classification, and robustness to random corruptions.
• Models trained with InfoDrop display shape-biased behavior, evidenced by qualitative saliency maps and sensitivity to shape information.
• InfoDrop can be combined with adversarial training to further enhance robustness with modest overhead.
• InfoDrop degrades less on source domains and improves target-domain performance in single- and multi-source domain generalization settings.
• Improvements are particularly pronounced in tasks emphasizing shapes (e.g., sketch-like domains) and in cross-domain transfer scenarios.