[Paper Review] Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks

Stefan Lee, Senthil Purushwalkam+3 more|arXiv (Cornell University)|Nov 19, 2015

Domain Adaptation and Few-Shot Learning203 citations

TL;DR

This paper proposes a unified framework for training diverse, high-performing deep neural network ensembles by introducing TreeNets—models with shared early layers and specialized deeper layers—and ensemble-aware losses that explicitly encourage diversity. It demonstrates that training under a Multiple Choice Learning (MCL) loss and using parameter sharing in TreeNets significantly improves ensemble accuracy, achieving up to 74.67% oracle accuracy on ILSVRC-AlexNet, outperforming standard ensembles.

ABSTRACT

Convolutional Neural Networks have achieved state-of-the-art performance on a wide range of tasks. Most benchmarks are led by ensembles of these powerful learners, but ensembling is typically treated as a post-hoc procedure implemented by averaging independently trained models with model variation induced by bagging or random initialization. In this paper, we rigorously treat ensembling as a first-class problem to explicitly address the question: what are the best strategies to create an ensemble? We first compare a large number of ensembling strategies, and then propose and evaluate novel strategies, such as parameter sharing (through a new family of models we call TreeNets) as well as training under ensemble-aware and diversity-encouraging losses. We demonstrate that TreeNets can improve ensemble performance and that diverse ensembles can be trained end-to-end under a unified loss, achieving significantly higher "oracle" accuracies than classical ensembles.

Motivation & Objective

• To treat deep neural network ensembling as a first-class problem rather than a post-hoc procedure.
• To investigate how parameter sharing, ensemble-aware losses, and diversity-encouraging training strategies affect ensemble performance.
• To develop a scalable distributed training framework for coupled ensembles that overcomes GPU memory and training time limitations.
• To evaluate the effectiveness of diversity in deep network ensembles across multiple datasets and architectures.
• To demonstrate that optimal ensembles are not classical independent models, but structured, shared architectures like TreeNets.

Proposed method

• Proposes TreeNets, a family of tree-structured deep networks that share early layers across ensemble members and allow controlled depth of parameter sharing.
• Introduces an ensemble-aware loss that minimizes the loss of the ensemble mean, encouraging collective performance.
• Develops a Multiple Choice Learning (MCL) loss that explicitly encourages diversity among ensemble members by treating them as experts on different subsets of predictions.
• Uses gradient accumulation across batches to enable larger networks to specialize under the MCL loss.
• Designs and implements MPI-Caffe, a distributed deep learning framework using MPI for cross-GPU communication, enabling model-parallel training of coupled ensembles.
• Applies communication layers (MPIBroadcast and MPIGather) in Caffe to allow distributed training of ensembles with shared or coupled components.

Experimental results

Research questions

• RQ1 Can parameter sharing in early layers improve ensemble performance while reducing model parameters?
• RQ2 Does training under an ensemble-aware loss that optimizes the ensemble mean improve generalization compared to independent training?
• RQ3 Can a diversity-encouraging loss like MCL significantly enhance oracle accuracy in deep network ensembles?
• RQ4 What is the optimal trade-off between parameter sharing and model independence in deep ensemble architectures?
• RQ5 Can distributed training frameworks like MPI-Caffe make large-scale, coupled ensemble training practical and efficient?

Key findings

• TreeNets with 1–2 shared initial layers achieve higher performance than classical independent ensembles, with optimal performance observed at intermediate sharing levels.
• Training under the MCL loss increases oracle accuracy to 72.67% when fine-tuning from a single network, compared to 56.90% for an identical initial ensemble.
• The MCL loss improves CIFAR10 ensemble-mean accuracy by 1% over standard ensembles when combined with cross-entropy loss.
• Ensembles trained with MCL loss achieve significantly higher oracle accuracies than classical ensembles, demonstrating the value of explicit diversity induction.
• Communication overhead in the MPI-Caffe framework is minimal—only 0.49% of forward-backward time is spent on communication, even for large layers like pool2 (36M floats).
• The study shows that diversity is most effective in high-level representations, while early layers benefit from weight sharing due to their generic feature learning.