[Paper Review] Deep Ensembles on a Fixed Memory Budget: One Wide Network or Several Thinner Ones?

Nadezhda Chirkova, Ekaterina Lobacheva+1 more|arXiv (Cornell University)|May 14, 2020

Advanced Neural Network Applications39 references6 citations

TL;DR

This paper investigates whether, under a fixed memory budget, performance is better achieved with a single wide neural network or an ensemble of thinner networks. It finds that splitting the memory into an ensemble of medium-width networks (Memory Split Advantage, MSA) consistently outperforms both a single wide network and very large ensembles of thin networks, with empirical gains up to 1.92% accuracy on CIFAR-100 using WideResNet-28-10 configurations.

ABSTRACT

One of the generally accepted views of modern deep learning is that increasing the number of parameters usually leads to better quality. The two easiest ways to increase the number of parameters is to increase the size of the network, e.g. width, or to train a deep ensemble; both approaches improve the performance in practice. In this work, we consider a fixed memory budget setting, and investigate, what is more effective: to train a single wide network, or to perform a memory split -- to train an ensemble of several thinner networks, with the same total number of parameters? We find that, for large enough budgets, the number of networks in the ensemble, corresponding to the optimal memory split, is usually larger than one. Interestingly, this effect holds for the commonly used sizes of the standard architectures. For example, one WideResNet-28-10 achieves significantly worse test accuracy on CIFAR-100 than an ensemble of sixteen thinner WideResNets: 80.6% and 82.52% correspondingly. We call the described effect the Memory Split Advantage and show that it holds for a variety of datasets and model architectures.

Motivation & Objective

• To investigate whether a fixed memory budget is better used for a single wide network or an ensemble of thinner networks.
• To determine the optimal memory split between model width and ensemble size for maximizing performance.
• To evaluate whether the Memory Split Advantage (MSA) holds across diverse datasets, architectures, and hyperparameter settings.
• To assess the impact of MSA on both classification accuracy and uncertainty estimation quality.

Proposed method

• The authors compare a single wide network against ensembles of thinner networks with identical total parameter counts across multiple datasets and architectures.
• They perform experiments on CIFAR-10, CIFAR-100, and IWSLT’14 using VGG, WideResNet, and Transformer models.
• Hyperparameters are tuned via Bayesian optimization and grid search to ensure fair comparison across configurations.
• Performance is evaluated using test accuracy and calibrated negative log-likelihood (NLL) for uncertainty estimation.
• The memory split is systematically varied by adjusting the number of networks (N) and their width (S), with total parameters held constant.
• The MSA effect is quantified by identifying the ensemble configuration (N, S) that maximizes test accuracy and minimizes NLL.

Experimental results

Research questions

• RQ1 Does an ensemble of medium-width networks outperform a single wide network when the total number of parameters is fixed?
• RQ2 Is there an optimal number of networks in an ensemble that maximizes performance under a fixed memory budget?
• RQ3 Does the Memory Split Advantage (MSA) hold across different datasets and model architectures?
• RQ4 How does the MSA effect compare in terms of uncertainty estimation quality, measured by calibrated NLL?
• RQ5 Does hyperparameter tuning strategy (Bayesian optimization vs. grid search) affect the observed MSA effect?

Key findings

• An ensemble of sixteen thinner WideResNets (each 80.6% accuracy) outperforms a single WideResNet-28-10 (82.52% accuracy), demonstrating a 1.92% gain on CIFAR-100.
• The Memory Split Advantage (MSA) holds even for small architecture configurations, not just large ones, indicating broad applicability.
• For the same total parameter count, ensembles of medium-width networks achieve lower calibrated test NLL than single wide networks, indicating better uncertainty estimation.
• The optimal ensemble size (N) for maximum performance is typically larger than one and increases with the total memory budget.
• Bayesian optimization of hyperparameters yields higher test accuracy than grid search for single models, but the MSA effect remains consistent across both tuning methods.
• The MSA effect is robust across multiple datasets (CIFAR-10, CIFAR-100) and architectures (VGG, WideResNet, Transformer), confirming its generalizability.