[Paper Review] Towards a Theoretical Framework of Out-of-Distribution Generalization

Haotian Ye, Chuanlong Xie+4 more|arXiv (Cornell University)|Jun 8, 2021

Domain Adaptation and Few-Shot LearningComputer Science60 references41 citations

TL;DR

The paper introduces a quantitative framework for OOD generalization using variation, informativeness, and expansion functions, derives OOD generalization bounds, and proposes a model selection criterion that improves OOD accuracy in experiments.

ABSTRACT

Generalization to out-of-distribution (OOD) data is one of the central problems in modern machine learning. Recently, there is a surge of attempts to propose algorithms that mainly build upon the idea of extracting invariant features. Although intuitively reasonable, theoretical understanding of what kind of invariance can guarantee OOD generalization is still limited, and generalization to arbitrary out-of-distribution is clearly impossible. In this work, we take the first step towards rigorous and quantitative definitions of 1) what is OOD; and 2) what does it mean by saying an OOD problem is learnable. We also introduce a new concept of expansion function, which characterizes to what extent the variance is amplified in the test domains over the training domains, and therefore give a quantitative meaning of invariant features. Based on these, we prove OOD generalization error bounds. It turns out that OOD generalization largely depends on the expansion function. As recently pointed out by Gulrajani and Lopez-Paz (2020), any OOD learning algorithm without a model selection module is incomplete. Our theory naturally induces a model selection criterion. Extensive experiments on benchmark OOD datasets demonstrate that our model selection criterion has a significant advantage over baselines.

Motivation & Objective

• Formalize OOD generalization as a relation between available and unseen domains using feature distributions.
• Introduce variation, informativeness, and expansion functions to quantify invariance and learnability.
• Derive upper and lower bounds on OOD generalization error based on expansion and feature variation.
• Propose a model selection criterion that balances validation accuracy with feature variation to improve OOD performance.
• Demonstrate the approach with experiments on standard OOD datasets and analyze the learnability of real-world OOD problems.

Proposed method

• Define variation and informativeness for one-dimensional features across domains using a distribution distance ρ.
• Introduce an expansion function s(·) to relate variation between available and unseen domains.
• Formulate learnability of OOD as the existence of (s(·), δ)-learnability given an informativeness threshold.
• Provide generalization bounds: err(f) ≤ O(s(Vsup(h, Eavail))^(α^2/(α+d)^2)) under listed regularity conditions.
• Specialize bounds for linear top models, showing potentially linear convergence: err(f) ≤ O(s(Vsup(h, Eavail))).
• Propose a model selection algorithm that optimizes Acc - r0 · V, combining validation accuracy with feature variation.

Experimental results

Research questions

• RQ1 How can OOD generalization be rigorously characterized when training and test domains differ?
• RQ2 What roles do feature variation and informativeness play in preserving invariance across unseen domains?
• RQ3 Can we derive quantitative bounds on OOD generalization error that depend on expansion functions and feature variation?
• RQ4 Can model selection be improved by considering both validation performance and feature variation to predict OOD performance?

Key findings

• The expansion function s(·) quantifies how feature variation grows from available to all domains and determines OOD difficulty.
• Ood generalization error is upper-bounded and lower-bounded by quantities tied to feature variation and informativeness, with bounds tightening as variation decreases.
• For informative features with low variation on available domains, OOD generalization improves, and the error can approach zero under certain conditions.
• A model selection criterion that combines validation accuracy with a variation penalty outperforms accuracy-only selection in several OOD benchmarks.
• Empirical analysis on Office-Home shows an identifiable expansion function; increasing δ ( informativeness threshold) reduces the expansion function and makes learning more feasible.
• Experiments on PACS and OfficeHome demonstrate that the proposed selection method yields higher OOD accuracy than validation-accuracy-based selection.