[Paper Review] On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law

Damien Teney, Kushal Kafle+4 more|arXiv (Cornell University)|May 19, 2020

Multimodal Machine Learning Applications41 references76 citations

TL;DR

The paper critiques the VQA-CP OOD benchmark, exposing three experimental flaws, showing simple baselines can beat state-of-the-art, and proposing guidelines to properly assess generalization with OOD testing.

ABSTRACT

Out-of-distribution (OOD) testing is increasingly popular for evaluating a machine learning system's ability to generalize beyond the biases of a training set. OOD benchmarks are designed to present a different joint distribution of data and labels between training and test time. VQA-CP has become the standard OOD benchmark for visual question answering, but we discovered three troubling practices in its current use. First, most published methods rely on explicit knowledge of the construction of the OOD splits. They often rely on ``inverting'' the distribution of labels, e.g. answering mostly 'yes' when the common training answer is 'no'. Second, the OOD test set is used for model selection. Third, a model's in-domain performance is assessed after retraining it on in-domain splits (VQA v2) that exhibit a more balanced distribution of labels. These three practices defeat the objective of evaluating generalization, and put into question the value of methods specifically designed for this dataset. We show that embarrassingly-simple methods, including one that generates answers at random, surpass the state of the art on some question types. We provide short- and long-term solutions to avoid these pitfalls and realize the benefits of OOD evaluation.

Motivation & Objective

• Motivate robust evaluation of generalization beyond training biases in VQA and similar tasks.
• Identify and articulate three practical flaws that undermine OOD benchmarks like VQA-CP.
• Demonstrate that embarrassingly-simple baselines can outperform complex methods on VQA-CP due to these flaws.
• Propose concrete recommendations to preserve the value of OOD evaluation and guide future benchmark design.

Proposed method

• Describe and evaluate simple baselines, including random predictions and inverted random predictions, to reveal how OOD exploitation can occur.
• Introduce and evaluate a range of baselines and simple regularization ideas (e.g., learned baselines, top-answer masking, random-image regularizers) to test their impact on OOD and in-domain performance.
• Analyze how the use of the OOD test set for model selection and retraining on in-domain splits distort the intended purpose of OOD benchmarks.
• Provide empirical evidence that high OOD performance can come at the cost of in-domain performance, especially under retraining on VQA-v2.

Experimental results

Research questions

• RQ1 How do current VQA-CP evaluation practices enable unintended exploitation of OOD splits?
• RQ2 What is the impact of using the OOD test set for model selection on reported generalization results?
• RQ3 How does retraining on in-domain splits (e.g., VQA-v2) affect OOD vs in-domain performance assessments?
• RQ4 What baselines or methodologies can reveal and mitigate the misuses of OOD benchmarks and guide better benchmark design?

Key findings

• Embarrassingly-simple methods, including random predictions, can surpass state-of-the-art on some VQA-CP question types by exploiting dataset biases.
• Using the OOD test set for model selection leads to adaptive overfitting and inflated OOD scores, masking true generalization.
• Retraining on VQA-v2 to evaluate in-domain performance hides substantial drops in OOD effectiveness, questioning the claimed generalization benefits.
• A simple random-inverted baseline can achieve very high OOD accuracy on yes/no and number questions, highlighting how yes/no/nb questions drive apparent gains without genuine understanding.
• A random-image regularizer can tune the trade-off between in-domain and OOD performance, but its benefits are often dominated by yes/no/nb questions and are not consistently beneficial across metrics.
• The authors argue for holding out in-domain validation data and focusing analysis on non yes/no/nb questions to better reflect true generalization and resist gaming of the benchmark.