[Paper Review] Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Mitchell Wortsman, Gabriel Ilharco+9 more|arXiv (Cornell University)|Mar 10, 2022

Domain Adaptation and Few-Shot Learning205 citations

TL;DR

Averaging weights from multiple independently fine-tuned models (model soups) often surpasses the best single model, achieving near-ensemble performance without extra inference cost. Greedy soup is highlighted as especially effective.

ABSTRACT

The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder. In this paper, we revisit the second step of this procedure in the context of fine-tuning large pre-trained models, where fine-tuned models often appear to lie in a single low error basin. We show that averaging the weights of multiple models fine-tuned with different hyperparameter configurations often improves accuracy and robustness. Unlike a conventional ensemble, we may average many models without incurring any additional inference or memory costs -- we call the results "model soups." When fine-tuning large pre-trained models such as CLIP, ALIGN, and a ViT-G pre-trained on JFT, our soup recipe provides significant improvements over the best model in a hyperparameter sweep on ImageNet. The resulting ViT-G model, which attains 90.94% top-1 accuracy on ImageNet, achieved a new state of the art. Furthermore, we show that the model soup approach extends to multiple image classification and natural language processing tasks, improves out-of-distribution performance, and improves zero-shot performance on new downstream tasks. Finally, we analytically relate the performance similarity of weight-averaging and logit-ensembling to flatness of the loss and confidence of the predictions, and validate this relation empirically. Code is available at https://github.com/mlfoundations/model-soups.

Motivation & Objective

• Motivate an alternative to selecting a single best fine-tuned model by leveraging weight averaging across hyperparameter diversities.
• Demonstrate that model soups improve accuracy and robustness on in-distribution and distribution-shift data.
• Show that greedy soups outperform uniform averaging and approach ensemble performance without extra inference costs.
• Explore applicability to vision (CLIP/ALIGN/VIT-G), and NLP tasks, including GLUE benchmarks.
• Provide analytical insight into when weight-averaging aligns with logit-ensembling and loss landscape flatness.

Proposed method

• Fine-tune large pre-trained models (e.g., CLIP, ALIGN, ViT-G) across broad hyperparameter sweeps.
• Form model soups by averaging weights of selected fine-tuned models (uniform soup) or via a greedy procedure that adds models only if validation accuracy improves.
• Compare soups to ensembles and best individual models under in-distribution and distribution shifts.
• Analyze two-model weight averaging to relate soup performance to logit-ensembling and loss flatness.
• Evaluate across image classification, distribution-shift datasets, and initial NLP GLUE tasks.
• Provide open-source code for model soups at the referenced GitHub repository.

Experimental results

Research questions

• RQ1 Does averaging the weights of independently fine-tuned models yield higher accuracy than selecting the best individual model?
• RQ2 How do uniform and greedy soups compare in accuracy and robustness across vision and NLP tasks?
• RQ3 How does model soup performance relate to ensemble performance and to the flatness of the loss landscape?
• RQ4 Do model soups extend to large vision transformers and language models beyond image classification?
• RQ5 What are the limitations in calibration and applicability of model soups across datasets and tasks?

Key findings

• Greedy soups often surpass the best individual model on ImageNet and distribution-shift datasets without extra training or inference cost.
• For CLIP and ALIGN fine-tuned on ImageNet, greedy soup improves over the best single model by 0.7–0.5 percentage points, respectively.
• A ViT-G/14 model soup on ImageNet reaches 90.94% top-1, approaching state-of-the-art with fewer FLOPs than prior methods.
• Model soups improve performance across image classification, distribution shifts, and some NLP GLUE tasks, though gains in calibration are limited.
• Analytical approximations show soup performance relates to loss landscape flatness and prediction confidence, with empirical validation.
• Greedy soups offer a practical alternative to ensembling when resources are constrained.