[Paper Review] Why do classifier accuracies show linear trends under distribution shift?

Horia Mania, Suvrit Sra|arXiv (Cornell University)|Dec 31, 2020

Machine Learning and Data Classification23 references4 citations

TL;DR

This paper explains why deep learning classifier accuracies exhibit approximately linear trends when evaluated under distribution shift, attributing the phenomenon to model similarity: when high-accuracy models correctly classify most examples that lower-accuracy models also get right. Under this assumption, linear relationships between accuracies on two distributions emerge unless the shift is large, offering insight into model robustness and generalization.

ABSTRACT

Recent studies of generalization in deep learning have observed a puzzling trend: accuracies of models on one data distribution are approximately linear functions of the accuracies on another distribution. We explain this trend under an intuitive assumption on model similarity, which was verified empirically in prior work. More precisely, we assume the probability that two models agree in their predictions is higher than what we can infer from their accuracy levels alone. Then, we show that a linear trend must occur when evaluating models on two distributions unless the size of the distribution shift is large. This work emphasizes the value of understanding model similarity, which can have an impact on the generalization and robustness of classification models.

Motivation & Objective

• Understand why deep learning models show approximately linear relationships between accuracies on original and shifted data distributions.
• Investigate the conditions under which such linear trends emerge, particularly in the context of distribution shift.
• Provide a theoretical foundation for the observed collinearity in model performance across datasets like CIFAR-10.1 and ImageNetV2.
• Explore how model similarity influences generalization and robustness, especially in light of recent distribution shift benchmarks.
• Identify scenarios where models break the linear trend, signaling potential for effective robustness beyond current model behavior.

Proposed method

• Formalize the problem using two data distributions, P (original) and Q (shifted), and define model accuracies μ_i and μ̃_i under each.
• Introduce an assumption based on model similarity: the probability that a high-accuracy model is correct while a lower-accuracy model is wrong is small.
• Derive theoretical bounds on the relationship between μ_i and μ̃_i using this assumption, showing that linear trends must emerge unless the distribution shift is large.
• Analyze the conditions under which a probit-scale linear fit provides a better fit than standard linear regression, linking it to model saturation and upper bounds.
• Use piecewise linear fitting to show that trends with two segments can fit data as well as probit-linear fits, suggesting broader applicability.
• Validate the theoretical findings empirically using ImageNet and CIFAR-10 models, comparing linear, probit, and piecewise fits.

Experimental results

Research questions

• RQ1 Why do classifier accuracies on two distributions exhibit approximately linear relationships under distribution shift?
• RQ2 Under what conditions does this linear trend break down, and when can it be expected to hold?
• RQ3 How does model similarity—specifically, agreement on correct predictions—explain the observed linear trends?
• RQ4 Why does a probit-scale transformation yield a better linear fit for ImageNet models, and is this phenomenon generalizable?
• RQ5 Can piecewise linear fits explain the data better than simple linear or probit-linear fits, and what does this imply about the underlying structure of model performance?

Key findings

• Under the assumption that high-accuracy models correctly classify most examples correctly classified by lower-accuracy models, a linear relationship between accuracies on two distributions must hold unless the distribution shift is large.
• The linear trend is robust to distribution shifts of moderate size, explaining the consistent observation of collinearity in benchmarks like CIFAR-10.1 and ImageNetV2.
• A probit-scale linear fit provides a superior fit for ImageNet models, which the authors link to model saturation and tight theoretical upper bounds.
• A piecewise linear fit with two segments achieves an R² of 0.997, matching the performance of the probit-linear fit (difference in R²: 4×10⁻⁴), suggesting piecewise trends may be more general than previously thought.
• The CLIP model, trained on a vastly larger dataset, breaks the linear trend on ImageNetV2, indicating it may be less similar to prior models and thus effectively robust, though this requires further analysis.
• When models are not similar to one another, new models can lie above or below the linear trend even under small distribution shifts, enabling effective robustness beyond what the trend predicts.