[Paper Review] Understanding the Failure Modes of Out-of-Distribution Generalization

Vaishnavh Nagarajan, Anders Andreassen+1 more|arXiv (Cornell University)|Oct 29, 2020

Domain Adaptation and Few-Shot LearningComputer Science59 references50 citations

TL;DR

The paper identifies two fundamental failure modes of ERM in OoD generalization—geometric and statistical skews induced by spurious correlations—and provides theoretical and empirical analysis on easy-to-learn tasks showing ERM reliance on spurious features. It also validates these modes on MNIST/CIFAR10 tasks and explores implications for neural networks.

ABSTRACT

Empirical studies suggest that machine learning models often rely on features, such as the background, that may be spuriously correlated with the label only during training time, resulting in poor accuracy during test-time. In this work, we identify the fundamental factors that give rise to this behavior, by explaining why models fail this way {\em even} in easy-to-learn tasks where one would expect these models to succeed. In particular, through a theoretical study of gradient-descent-trained linear classifiers on some easy-to-learn tasks, we uncover two complementary failure modes. These modes arise from how spurious correlations induce two kinds of skews in the data: one geometric in nature, and another, statistical in nature. Finally, we construct natural modifications of image classification datasets to understand when these failure modes can arise in practice. We also design experiments to isolate the two failure modes when training modern neural networks on these datasets.

Motivation & Objective

• Define the domain generalization setting and the ERM baseline.
• Identify when ERM fails due to spurious correlations even in easy-to-learn tasks.
• Theoretically isolate two failure modes: geometric and statistical skews.
• Empirically validate the failure modes on MNIST/CIFAR10 and neural nets.
• Provide dataset modifications and experiments to isolate the failure mechanisms.

Proposed method

• Model ERM and max-margin training on easy-to-learn tasks with fully predictive invariant features.
• Introduce constraints to define easy-to-learn domain generalization tasks (fully predictive invariant features, identical invariant distribution, conditional independence, two-valued spurious features, identity Phi).
• Theoretically derive bounds showing how spurious features influence max-margin solutions under geometric skews (Theorem sketches).
• Analyze finite-time gradient-descent dynamics on logistic/exponential loss to reveal statistical skews (convergence bounds).
• Empirically demonstrate geometric skew via increasing-norm behavior and spurious-feature reliance on MNIST/CIFAR10-like tasks.
• Describe and reference code for reproducing experiments.

Experimental results

Research questions

• RQ1 Why does ERM fail to generalize under OoD despite fully predictive invariant features?
• RQ2 What are the fundamental geometric and statistical factors that drive reliance on spurious features?
• RQ3 Can easy-to-learn tasks exhibit ERM failures, and under what constraints?
• RQ4 How do these failure modes manifest in neural networks beyond linear models?
• RQ5 What empirical evidence supports the proposed failure modes on standard datasets?

Key findings

• ERM can rely on spurious features even when invariant features fully determine the label in easy-to-learn tasks.
• Two complementary failure modes are identified: geometric skews that emerge from data geometry and increase in max-margin norm with more data, and statistical skews related to finite-time gradient descent dynamics.
• As spurious correlations strengthen, the geometric skew pushes the max-margin classifier to use the spurious feature as a shortcut, increasing vulnerability to test-time shifts.
• Finite-time gradient descent shows spurious-feature convergence that slows with higher spurious correlation, revealing a statistical skew in training dynamics.
• Empirical results on MNIST- and CIFAR10-based tasks and on networks like FNNs and ResNets support the presence of these skews across architectures.
• The paper also discusses scenarios where spurious correlations do not apply and provides a broader geometric interpretation of OoD failures.