[Paper Review] WILDS: A Benchmark of in-the-Wild Distribution Shifts

Pang Wei Koh|CaltechAUTHORS (California Institute of Technology)|Dec 14, 2020

AI in cancer detection393 references286 citations

TL;DR

WILDS presents a curated benchmark of 10 real-world distribution-shift datasets across diverse modalities, documenting how standard and baseline robustness methods underperform on out-of-distribution data, with an open-source package and leaderboards to facilitate method development.

ABSTRACT

Distribution shifts -- where the training distribution differs from the test distribution -- can substantially degrade the accuracy of machine learning (ML) systems deployed in the wild. Despite their ubiquity in the real-world deployments, these distribution shifts are under-represented in the datasets widely used in the ML community today. To address this gap, we present WILDS, a curated benchmark of 10 datasets reflecting a diverse range of distribution shifts that naturally arise in real-world applications, such as shifts across hospitals for tumor identification; across camera traps for wildlife monitoring; and across time and location in satellite imaging and poverty mapping. On each dataset, we show that standard training yields substantially lower out-of-distribution than in-distribution performance. This gap remains even with models trained by existing methods for tackling distribution shifts, underscoring the need for new methods for training models that are more robust to the types of distribution shifts that arise in practice. To facilitate method development, we provide an open-source package that automates dataset loading, contains default model architectures and hyperparameters, and standardizes evaluations. Code and leaderboards are available at https://wilds.stanford.edu.

Motivation & Objective

• Motivate and quantify the impact of real-world distribution shifts on ML models.
• Provide a diverse, realistic set of domain generalization and subpopulation shift benchmarks.
• Offer an open-source loading/evaluation package and leaderboards to standardize progress in robustness.

Proposed method

• Curate 10 real-world datasets capturing domain generalization and subpopulation shifts across multiple modalities.
• Define train/test splits with domain annotations to enable domain-aware learning.
• Evaluate standard training and existing shift-robust baselines to quantify performance gaps between ID and OOD.
• Provide an open-source Wilds package that automates data loading, model baselines, and evaluation pipelines.
• Establish a public leaderboard to track robustness against distribution shifts.

Experimental results

Research questions

• RQ1 To what extent do standard training procedures degrade when facing real-world distribution shifts across domains and subpopulations?
• RQ2 Do existing methods for distribution shift robustness close the performance gap on Wilds datasets?
• RQ3 Can domain annotations be leveraged to improve robustness across unseen domains or subpopulations?
• RQ4 How do robustness gaps manifest across diverse data modalities and tasks?

Key findings

• Across all datasets, training performance degrades on out-of-distribution data relative to in-distribution evaluation.
• Baseline methods for distribution shifts show persistent gaps, underscoring the need for new robustness approaches.
• The benchmark includes domains such as hospitals, camera traps, satellite imagery, and user/region variants to reflect real-world shifts.
• An open-source package standardizes dataset loading, models, hyperparameters, and evaluations, with a public leaderboard to track progress.