[Paper Review] Exploring the Limits of Large Scale Pre-training

Samira Abnar, Mostafa Dehghani+2 more|arXiv (Cornell University)|Oct 5, 2021

Multimodal Machine Learning Applications52 references35 citations

TL;DR

The paper shows that downstream image recognition performance saturates as upstream accuracy increases, even with large-scale pre-training across Vision Transformers, MLP-Mixers, and ResNets, and provides a power-law model to predict downstream performance from upstream accuracy.

ABSTRACT

Recent developments in large-scale machine learning suggest that by scaling up data, model size and training time properly, one might observe that improvements in pre-training would transfer favorably to most downstream tasks. In this work, we systematically study this phenomena and establish that, as we increase the upstream accuracy, the performance of downstream tasks saturates. In particular, we investigate more than 4800 experiments on Vision Transformers, MLP-Mixers and ResNets with number of parameters ranging from ten million to ten billion, trained on the largest scale of available image data (JFT, ImageNet21K) and evaluated on more than 20 downstream image recognition tasks. We propose a model for downstream performance that reflects the saturation phenomena and captures the nonlinear relationship in performance of upstream and downstream tasks. Delving deeper to understand the reasons that give rise to these phenomena, we show that the saturation behavior we observe is closely related to the way that representations evolve through the layers of the models. We showcase an even more extreme scenario where performance on upstream and downstream are at odds with each other. That is, to have a better downstream performance, we need to hurt upstream accuracy.

Motivation & Objective

• Investigate how improvements in upstream pre-training accuracy transfer to downstream image recognition tasks across diverse architectures and datasets.
• Quantify saturation phenomena in downstream performance as upstream accuracy increases.
• Develop a robust predictive model (power-law) linking upstream and downstream accuracies that accounts for nonlinearity and data/sample biases.
• Identify architectural and hyper-parameter factors that influence downstream transfer and reveal where downstream tasks may not benefit from further upstream gains.

Proposed method

• Aggregate and analyze over 4800 experiments across Vision Transformers, MLP-Mixers, and ResNets pre-trained on JFT or ImageNet21K and evaluated on 20+ downstream tasks.
• Model downstream performance as a function of upstream accuracy using a power-law form with an irreducible error term.
• Use convex hull (best-case) fitting to mitigate sampling bias and capture the maximal DS performance for given US accuracy.
• Conduct controlled scaling experiments varying model size, data size, and compute to assess their impact on DS performance through US accuracy.
• Inspect representation evolution across layers to explain observed saturation phenomena.
• Explore cases where upstream and downstream performances diverge due to head hyper-parameter settings that push information from the head to lower layers.

Experimental results

Research questions

• RQ1 Does improving upstream accuracy through scaling or hyper-parameter changes translate to proportional gains in downstream accuracy across tasks?
• RQ2 Is downstream performance universally increasing with upstream gains, or does saturation occur?
• RQ3 Can downstream performance be reliably predicted from upstream accuracy using a simple model across architectures and tasks?
• RQ4 What factors (architecture, data, compute, hyper-parameters) modulate the DS-US relationship and saturation level?
• RQ5 Why can upstream and downstream performances diverge under certain hyper-parameter configurations?

Key findings

• Downstream accuracy saturates as upstream accuracy increases, with DS rarely reaching 100% even at high US accuracy.
• A power-law relationship with an irreducible error term effectively models DS vs US performance across tasks and shots.
• The best model for a downstream task can vary across tasks even at similar upstream accuracies; there is no one-model-fits-all.
• Fitting the power-law to the convex hull of results provides robust DS predictions, being less sensitive to sample density.
• US accuracy provides strong predictive power for DS accuracy; conditioning on US accuracy reduces the incremental value of varying model size, data size, or compute for DS performance.
• Hyper-parameter choices, including head weight decay and learning rate, can cause DS and US performance to diverge, explained by information being pushed from the head to lower layers.