[Paper Review] The Pitfalls of Simplicity Bias in Neural Networks

Harshay Shah, Kaustav Tamuly+3 more|arXiv (Cornell University)|Jun 13, 2020

Adversarial Robustness in Machine Learning81 references88 citations

TL;DR

The paper formalizes Simplicity Bias (SB) in SGD-trained neural networks, showing that networks can over-rely on the simplest predictive feature, leading to brittleness, poor robustness, and even degraded generalization, and it provides datasets and experiments demonstrating these pitfalls across architectures and training methods.

ABSTRACT

Several works have proposed Simplicity Bias (SB)---the tendency of standard training procedures such as Stochastic Gradient Descent (SGD) to find simple models---to justify why neural networks generalize well [Arpit et al. 2017, Nakkiran et al. 2019, Soudry et al. 2018]. However, the precise notion of simplicity remains vague. Furthermore, previous settings that use SB to theoretically justify why neural networks generalize well do not simultaneously capture the non-robustness of neural networks---a widely observed phenomenon in practice [Goodfellow et al. 2014, Jo and Bengio 2017]. We attempt to reconcile SB and the superior standard generalization of neural networks with the non-robustness observed in practice by designing datasets that (a) incorporate a precise notion of simplicity, (b) comprise multiple predictive features with varying levels of simplicity, and (c) capture the non-robustness of neural networks trained on real data. Through theory and empirics on these datasets, we make four observations: (i) SB of SGD and variants can be extreme: neural networks can exclusively rely on the simplest feature and remain invariant to all predictive complex features. (ii) The extreme aspect of SB could explain why seemingly benign distribution shifts and small adversarial perturbations significantly degrade model performance. (iii) Contrary to conventional wisdom, SB can also hurt generalization on the same data distribution, as SB persists even when the simplest feature has less predictive power than the more complex features. (iv) Common approaches to improve generalization and robustness---ensembles and adversarial training---can fail in mitigating SB and its pitfalls. Given the role of SB in training neural networks, we hope that the proposed datasets and methods serve as an effective testbed to evaluate novel algorithmic approaches aimed at avoiding the pitfalls of SB.

Motivation & Objective

• Define a precise, tunable notion of feature simplicity and predictive power to study SB.
• Design modular synthetic and image-based datasets that combine simple and complex predictive features.
• Theoretically and experimentally demonstrate extreme SB across architectures and optimizers.
• Link SB to non-robustness, distribution shifts, and adversarial vulnerability, and evaluate common remedies.

Proposed method

• Introduce a formal notion of feature simplicity via the minimum number of linear pieces in a decision boundary.
• Construct multi-dimensional synthetic datasets (e.g., LMS-k, L̂MS-k, MS-(5,7), MS-5) and MNIST-CIFAR images that combine simple and complex features.
• Prove that one-hidden-layer ReLU networks trained with SGD exhibit SB on the LSN dataset.
• Empirically show SB across FCN, CNN, and GRU models, and under different optimizers and regularization methods.
• Analyze robustness, confidence estimation, and generalization under SB, including UAP transferability analyses.

Experimental results

Research questions

• RQ1 Does SGD-trained models exhibit a bias toward the simplest predictive feature when multiple predictive features are present?
• RQ2 How extreme is SB across architectures and training settings, and does it persist when simple features have lower margins?
• RQ3 What are the implications of SB for robustness, confidence estimates, and generalization under distribution shifts or adversarial perturbations?
• RQ4 Do ensembles or adversarial training mitigate SB and its pitfalls?

Key findings

• SB can be extreme: neural networks can rely exclusively on the simplest feature and ignore complex predictive features.
• Extreme SB correlates with poor robustness to adversarial perturbations and distribution shifts.
• SB can also hurt generalization when the simple feature is less predictive than complex features.
• Ensembles and adversarial training do not reliably mitigate SB on the proposed datasets.