[Paper Review] Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization

Devansh Arpit, Huan Wang+2 more|arXiv (Cornell University)|Oct 21, 2021

Domain Adaptation and Few-Shot Learning45 references47 citations

TL;DR

The paper introduces a simple moving-average model protocol (SMA) and an ensemble of moving-average models (EoA) to improve domain generalization, stabilize out-domain performance, and enhance model selection reliability without hyperparameter tuning. It shows EoA outperforms ERM baselines and prior DG methods across DomainBed benchmarks, with theoretical bias-variance insights.

ABSTRACT

In Domain Generalization (DG) settings, models trained independently on a given set of training domains have notoriously chaotic performance on distribution shifted test domains, and stochasticity in optimization (e.g. seed) plays a big role. This makes deep learning models unreliable in real world settings. We first show that this chaotic behavior exists even along the training optimization trajectory of a single model, and propose a simple model averaging protocol that both significantly boosts domain generalization and diminishes the impact of stochasticity by improving the rank correlation between the in-domain validation accuracy and out-domain test accuracy, which is crucial for reliable early stopping. Taking advantage of our observation, we show that instead of ensembling unaveraged models (that is typical in practice), ensembling moving average models (EoA) from independent runs further boosts performance. We theoretically explain the boost in performance of ensembling and model averaging by adapting the well known Bias-Variance trade-off to the domain generalization setting. On the DomainBed benchmark, when using a pre-trained ResNet-50, this ensemble of averages achieves an average of $68.0\%$, beating vanilla ERM (w/o averaging/ensembling) by $\sim 4\%$, and when using a pre-trained RegNetY-16GF, achieves an average of $76.6\%$, beating vanilla ERM by $6\%$. Our code is available at https://github.com/salesforce/ensemble-of-averages.

Motivation & Objective

• Investigate why optimization dynamics cause chaotic in-domain vs out-domain performance in DG.
• Propose a simple, hyperparameter-free moving-average protocol to stabilize and improve DG performance.
• Show that ensembling moving-average models (EoA) yields further gains over traditional ensembles.
• Provide theoretical insight by adapting Bias-Variance decomposition to domain generalization.
• Benchmark SMA and EoA against ERM and state-of-the-art DG methods on DomainBed across multiple backbones.

Proposed method

• Define online (unaveraged) models and moving-average (MA) models during training.
• Propose tail-averaging: start SMA after t0 iterations and maintain it to the end, using SMA for validation and testing.
• Show that SMA improves rank correlation between in-domain validation accuracy and out-domain test accuracy for reliable model selection.
• Demonstrate that an ensemble of moving-average models (EoA) outperforms ensembles of unaveraged models.
• Provide bias-variance based theoretical justification for why averaging/ensembling reduces out-domain error.
• Benchmark with three pre-trained backbones (ResNet-50, ResNeXt-50 32x4d, RegNetY-16GF) on DomainBed datasets.

Experimental results

Research questions

• RQ1 Does a hyperparameter-free moving-average protocol (SMA) reduce out-domain instability and improve reliable early stopping in DG?
• RQ2 Can ensembling moving-average models (EoA) outperform ensembles of online models and single models in DG benchmarks?
• RQ3 How does the Bias-Variance perspective adapt to DG to explain gains from model averaging and ensembling?
• RQ4 Do larger pre-trained models and datasets amplify the benefits of SMA and EoA in DG?
• RQ5 How does SMA affect rank correlation between in-domain validation and out-domain test performance across datasets?

Key findings

• SMA improves the rank correlation between in-domain validation accuracy and out-domain test accuracy, aiding reliable model selection.
• EoA consistently outperforms ensembles of unaveraged models across multiple datasets and backbone architectures.
• On DomainBed with ResNet-50 pre-trained on ImageNet, EoA achieves 68.0 average accuracy vs 64.0 for ERM, surpassing SWAD by 1.1 percentage points.
• With ResNeXt-50 32x4d, EoA reaches 72.7 average vs 67.7 for ERM, demonstrating larger gains with bigger/pretrained models.
• With RegNetY-16GF, EoA attains 76.6 average vs 70.6 for ERM, indicating substantial improvements as model size and pre-training scale increase.
• Theoretical framing ties the gains to a bias-dominated ensemble outcome in DG, contrasting with flat-minima explanations.