[Paper Review] In Search of Lost Domain Generalization

Ishaan Gulrajani, David López-Paz|arXiv (Cornell University)|Jul 2, 2020

Domain Adaptation and Few-Shot LearningComputer Science84 references85 citations

TL;DR

The paper argues that domain generalization methods must include a model selection strategy and shows that, with careful implementation and strong baselines, empirical risk minimization (ERM) matches or exceeds state-of-the-art across multiple datasets. It also introduces DomainBed, a PyTorch testbed for fair, reproducible evaluation of DG methods.

ABSTRACT

The goal of domain generalization algorithms is to predict well on distributions different from those seen during training. While a myriad of domain generalization algorithms exist, inconsistencies in experimental conditions -- datasets, architectures, and model selection criteria -- render fair and realistic comparisons difficult. In this paper, we are interested in understanding how useful domain generalization algorithms are in realistic settings. As a first step, we realize that model selection is non-trivial for domain generalization tasks. Contrary to prior work, we argue that domain generalization algorithms without a model selection strategy should be regarded as incomplete. Next, we implement DomainBed, a testbed for domain generalization including seven multi-domain datasets, nine baseline algorithms, and three model selection criteria. We conduct extensive experiments using DomainBed and find that, when carefully implemented, empirical risk minimization shows state-of-the-art performance across all datasets. Looking forward, we hope that the release of DomainBed, along with contributions from fellow researchers, will streamline reproducible and rigorous research in domain generalization.

Motivation & Objective

• Assess realism of domain generalization evaluations under varying datasets, models, and selection criteria.
• Investigate how model selection strategies affect DG method performance.
• Provide a standardized, reusable testbed for DG experiments to improve reproducibility.
• Promote stronger baselines and fair comparisons in domain generalization research.

Proposed method

• Review and articulate the challenge of model selection in domain generalization.
• Implement and compare nine DG algorithms on seven multi-domain datasets using DomainBed.
• Evaluate three model selection criteria (training-domain validation, leave-one-domain-out, and test-domain oracle).
• Conduct 20 random hyperparameter searches per algorithm/dataset configuration with three independent runs.
• Report results with strict averaging over runs and provide standard errors.
• Release DomainBed to streamline adding new algorithms/datasets and enable end-to-end experimentation.

Experimental results

Research questions

• RQ1 How do different model selection strategies affect domain generalization performance?
• RQ2 Do DG algorithms consistently outperform a strong ERM baseline under realistic, standardized evaluation conditions?
• RQ3 Can a standardized testbed (DomainBed) enable fairer, more reproducible domain generalization research?

Key findings

• ERM achieves state-of-the-art performance across the evaluated datasets when implemented with modern architectures, data augmentation, and careful hyperparameter tuning.
• No DG algorithm consistently outperforms ERM by more than a small margin when conditions are equal across datasets and configurations.
• Model selection strategy substantially influences DG results; training-domain validation often outperforms leave-one-domain-out, while oracle (test-domain) selection shows headroom for improvement.
• DomainBed provides a scalable, reproducible framework to run DG experiments, where adding new algorithms or datasets is lightweight.
• A combination of larger networks (ResNet-50), aggressive data augmentation, and thorough hyperparameter search largely accounts for ERM’s strong performance.