[Paper Review] Deep Ensembles: A Loss Landscape Perspective

Stanislav Fort, Huiyi Hu+1 more|arXiv (Cornell University)|Dec 5, 2019

Generative Adversarial Networks and Image Synthesis35 references347 citations

TL;DR

The paper shows that random initialization explores distinct function-space modes, while subspace sampling within a single trajectory yields similar functions; random ensembles outperform subspace methods in diversity–accuracy trade-offs.

ABSTRACT

Deep ensembles have been empirically shown to be a promising approach for improving accuracy, uncertainty and out-of-distribution robustness of deep learning models. While deep ensembles were theoretically motivated by the bootstrap, non-bootstrap ensembles trained with just random initialization also perform well in practice, which suggests that there could be other explanations for why deep ensembles work well. Bayesian neural networks, which learn distributions over the parameters of the network, are theoretically well-motivated by Bayesian principles, but do not perform as well as deep ensembles in practice, particularly under dataset shift. One possible explanation for this gap between theory and practice is that popular scalable variational Bayesian methods tend to focus on a single mode, whereas deep ensembles tend to explore diverse modes in function space. We investigate this hypothesis by building on recent work on understanding the loss landscape of neural networks and adding our own exploration to measure the similarity of functions in the space of predictions. Our results show that random initializations explore entirely different modes, while functions along an optimization trajectory or sampled from the subspace thereof cluster within a single mode predictions-wise, while often deviating significantly in the weight space. Developing the concept of the diversity--accuracy plane, we show that the decorrelation power of random initializations is unmatched by popular subspace sampling methods. Finally, we evaluate the relative effects of ensembling, subspace based methods and ensembles of subspace based methods, and the experimental results validate our hypothesis.

Motivation & Objective

• Investigate why deep ensembles formed from random initializations perform well in accuracy and uncertainty.
• Analyze loss landscapes to understand diversity of functions across different training trajectories.
• Compare random-initialization ensembles with subspace-based Bayesian approximations in terms of diversity and accuracy.
• Examine dataset shift robustness and the diversity–accuracy trade-off across methods.

Proposed method

• Train multiple neural networks from different random initializations to form ensembles.
• Analyze weight-space and function-space similarities across checkpoints and trajectories.
• Construct and compare subspaces (random subspace, dropout, diagonal Gaussian, low-rank Gaussian) around each trajectory.
• Use t-SNE on prediction vectors to visualize function-space diversity.
• Evaluate diversity–accuracy trade-offs and ensemble performance on CIFAR-10/100 and ImageNet, including corruptions and OOD data.
• Assess ensemble vs subspace methods under dataset shift using CIFAR-10-C and ImageNet-C.

Experimental results

Research questions

• RQ1 Do random initializations sample different function-space modes despite similar weight-space trajectories?
• RQ2 How do subspace sampling methods compare to independent ensembles in diversity and accuracy?
• RQ3 Can subspace-based approaches provide complementary benefits to ensembling, especially under dataset shift?
• RQ4 What is the relationship between diversity in function space and robustness to corruption or OOD inputs?

Key findings

• Checkpoints along a single trajectory are similar in both weight space and function space.
• Functions from different random initializations are diverse in function space but not in weight space.
• Subspace sampling methods produce functions close to their origin trajectory in function space and do not reach the diversity of independent optima.
• Independently trained ensembles achieve a better diversity–accuracy trade-off than subspace methods, and gains from ensembling grow with ensemble size.
• Ensembles and subspace methods are complementary, offering improved performance and uncertainty estimates, particularly under dataset shift (CIFAR-10-C, ImageNet-C).
• Jensen-Shannon divergence between predictions is highest for independent random initializations and much lower for within-trajectory subspaces, especially under corruption.