[Paper Review] Accounting for Variance in Machine Learning Benchmarks

Xavier Bouthillier, Pierre Delaunay+15 more|arXiv (Cornell University)|Mar 1, 2021

Adversarial Robustness in Machine Learning55 references40 citations

TL;DR

The paper formalizes variance sources in ML benchmarks (data sampling, initialization, augmentation, and hyperparameter optimization), shows their substantial impact on comparative claims, and proposes strategies (randomization and resampling) to reliably detect improvements, including a counter-intuitive 51x compute-cost reduction when accounting for more variation.

ABSTRACT

Strong empirical evidence that one machine-learning algorithm A outperforms another one B ideally calls for multiple trials optimizing the learning pipeline over sources of variation such as data sampling, data augmentation, parameter initialization, and hyperparameters choices. This is prohibitively expensive, and corners are cut to reach conclusions. We model the whole benchmarking process, revealing that variance due to data sampling, parameter initialization and hyperparameter choice impact markedly the results. We analyze the predominant comparison methods used today in the light of this variance. We show a counter-intuitive result that adding more sources of variation to an imperfect estimator approaches better the ideal estimator at a 51 times reduction in compute cost. Building on these results, we study the error rate of detecting improvements, on five different deep-learning tasks/architectures. This study leads us to propose recommendations for performance comparisons.

Motivation & Objective

• Quantify how various sources of randomness affect benchmark performance in ML pipelines.
• Model the full benchmarking process including hyperparameter optimization to understand its impact on performance estimates.
• Propose practical recommendations to improve the reliability of benchmark comparisons under limited compute budgets.
• Assess how to detect genuine improvements between algorithms under benchmark variance.

Proposed method

• Extend the Hothorn et al. (2005) framework to model variance from data, initialization, augmentation, and hyperparameter optimization in ML pipelines.
• Represent the complete learning pipeline as a stochastic process with random variations xiH and xiO that affect Opt(Stv, lambda).
• Use bootstrap sampling to quantify data-sampling variance and compare it to a binomial model for classification accuracy.
• Evaluate variance across multiple hyperparameter optimization methods (random search, grid search, Bayesian optimization) under controlled seeds and budgets.
• Analyze how estimators of average performance mu change when incorporating full variance versus partial variance, and derive implications for benchmark design.

Experimental results

Research questions

• RQ1 What are the main sources of uncontrolled variance in ML benchmarks beyond data sampling?
• RQ2 How does hyperparameter optimization contribute to variance in benchmark performance compared to other sources like initialization or data augmentation?
• RQ3 How does accounting for multiple sources of variation affect the reliability of detecting improvements between algorithms A and B?
• RQ4 What practical recommendations best improve the precision and fairness of benchmark conclusions under limited compute budgets?

Key findings

• Data sampling variance, hyperparameter optimization, and augmentation choices significantly influence benchmark outcomes.
• Bootstrapping data reveals it as the dominant source of variance, with initialization and SGD visit order contributing less but non-negligible portions.
• Hyperparameter optimization variance is substantial and can be comparable to initialization variance across studied cases.
• Incorporating more sources of variation reduces the standard error of the performance estimator, improving sensitivity to small improvements.
• The observed variance in published improvements is of the same order as the typical gains reported, underscoring the need to account for variance when benchmarking.
• A 51× reduction in compute cost is reported when accounting for additional sources of variation, approaching the ideal estimator.