[Paper Review] Understanding and Testing Generalization of Deep Networks on Out-of-Distribution Data

Rui Hu, Jitao Sang+3 more|arXiv (Cornell University)|Nov 17, 2021

Anomaly Detection Techniques and Applications50 references4 citations

TL;DR

This paper proposes a novel OOD test paradigm to evaluate deep networks' generalization on out-of-distribution data by identifying failures in standard in-distribution (ID) testing. It introduces a unified categorization of three distribution shifts—marginal, conditional, and joint—and demonstrates that ID tests fail due to spurious correlations; the proposed OOD testing identifies harmful irrelevant features, enabling targeted model debugging and improved practical performance.

ABSTRACT

Deep network models perform excellently on In-Distribution (ID) data, but can significantly fail on Out-Of-Distribution (OOD) data. While developing methods focus on improving OOD generalization, few attention has been paid to evaluating the capability of models to handle OOD data. This study is devoted to analyzing the problem of experimental ID test and designing OOD test paradigm to accurately evaluate the practical performance. Our analysis is based on an introduced categorization of three types of distribution shifts to generate OOD data. Main observations include: (1) ID test fails in neither reflecting the actual performance of a single model nor comparing between different models under OOD data. (2) The ID test failure can be ascribed to the learned marginal and conditional spurious correlations resulted from the corresponding distribution shifts. Based on this, we propose novel OOD test paradigms to evaluate the generalization capacity of models to unseen data, and discuss how to use OOD test results to find bugs of models to guide model debugging.

Motivation & Objective

• Address the gap in evaluating deep networks on out-of-distribution (OOD) data, as standard in-distribution (ID) testing fails to reflect real-world performance.
• Investigate why ID test results do not generalize to OOD scenarios, particularly under distribution shift.
• Develop a practical OOD testing framework to evaluate and debug models based on their OOD generalization capability.
• Enable identification of critical irrelevant features ("bugs") that undermine model performance in real-world deployment.

Proposed method

• Introduce a unified categorization of three types of distribution shift: marginal, conditional, and joint, to simulate OOD data generation.
• Construct ID and OOD training/test sets on CelebA and Colored MNIST datasets under controlled distribution shift settings.
• Analyze performance differences between ID and OOD settings to diagnose failures in standard ID evaluation.
• Identify two types of spurious correlations—marginal (irrelevant features used) and conditional (incorrect feature-label correlations)—as root causes of ID test failure.
• Propose OOD test paradigms for both known and unknown irrelevant features, using controlled distribution shifts at multiple levels.
• Integrate OOD test results into a closed-loop workflow for model debugging by identifying which irrelevant features critically degrade performance.

Experimental results

Research questions

• RQ1 Why do standard in-distribution (ID) tests fail to reflect the true out-of-distribution (OOD) performance of deep networks?
• RQ2 What are the underlying causes of ID test failure in OOD generalization scenarios?
• RQ3 How can we design a reliable OOD test paradigm to evaluate and compare model generalization on unseen data?
• RQ4 Can OOD test results be used to identify and debug harmful irrelevant features in deep learning models?
• RQ5 What role do marginal and conditional spurious correlations play in undermining OOD generalization?

Key findings

• ID testing fails to reflect actual OOD performance because models learn spurious correlations due to distribution shifts, leading to misleading performance estimates.
• Marginal and conditional spurious correlations—resulting from shifts in irrelevant feature distributions and incorrect label dependencies—directly cause ID test failure.
• Experiments show that models trained under ID settings exhibit significant performance drops on OOD data, especially when irrelevant features like color or texture are misused.
• The proposed OOD test paradigms successfully detect performance degradation caused by specific irrelevant features, such as hair color in gender classification tasks.
• By analyzing performance decline across different levels and types of distribution shifts, the OOD test identifies which features are critical bugs, enabling targeted data or model interventions.
• The OOD testing workflow enables a closed-loop between testing and debugging, guiding model improvements through feature-specific diagnostics.