[Paper Review] Efficient Diversity-Driven Ensemble for Deep Neural Networks

Wentao Zhang, Jiawei Jiang+2 more|arXiv (Cornell University)|Dec 26, 2021

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Efficient Diversity-Driven Ensemble (EDDE), a novel method that accelerates training and enhances diversity in deep neural network ensembles by selectively transferring generic knowledge from pre-trained models and optimizing with a diversity-driven loss function. EDDE achieves state-of-the-art ensemble accuracy with lower training cost than existing methods on both CV and NLP tasks.

ABSTRACT

The ensemble of deep neural networks has been shown, both theoretically and empirically, to improve generalization accuracy on the unseen test set. However, the high training cost hinders its efficiency since we need a sufficient number of base models and each one in the ensemble has to be separately trained. Lots of methods are proposed to tackle this problem, and most of them are based on the feature that a pre-trained network can transfer its knowledge to the next base model and then accelerate the training process. However, these methods suffer a severe problem that all of them transfer knowledge without selection and thus lead to low diversity. As the effect of ensemble learning is more pronounced if ensemble members are accurate and diverse, we propose a method named Efficient Diversity-Driven Ensemble (EDDE) to address both the diversity and the efficiency of an ensemble. To accelerate the training process, we propose a novel knowledge transfer method which can selectively transfer the previous generic knowledge. To enhance diversity, we first propose a new diversity measure, then use it to define a diversity-driven loss function for optimization. At last, we adopt a Boosting-based framework to combine the above operations, such a method can also further improve diversity. We evaluate EDDE on Computer Vision (CV) and Natural Language Processing (NLP) tasks. Compared with other well-known ensemble methods, EDDE can get highest ensemble accuracy with the lowest training cost, which means it is efficient in the ensemble of neural networks.

Motivation & Objective

• To address the inefficiency of existing ensemble methods that suffer from high training cost and low diversity due to unselective knowledge transfer.
• To improve ensemble generalization by simultaneously maximizing accuracy and diversity of base models under a limited training budget.
• To develop a knowledge transfer mechanism that selectively transfers generic features from pre-trained models to accelerate convergence without sacrificing diversity.
• To introduce a new diversity measure and a diversity-driven loss function that explicitly optimize for negative correlation among base models.
• To integrate a Boosting-based framework to further enhance diversity and improve ensemble performance.

Proposed method

• Proposes a new diversity measure based on the negative correlation of soft label predictions across base models.
• Introduces a diversity-driven loss function that combines standard cross-entropy loss with a regularization term promoting negative correlation in model outputs.
• Designs a selective knowledge transfer strategy that transfers only generic features (from lower layers) of pre-trained models to accelerate training of subsequent base models.
• Employs a Boosting-based training framework where each base model is trained sequentially with knowledge transfer and diversity optimization.
• Uses a hyperparameter γ to balance the trade-off between accuracy and diversity in the loss function, enabling controlled optimization.
• Applies the method to both computer vision (CIFAR-100, ImageNet) and natural language processing (GLUE benchmark) tasks for evaluation.

Experimental results

Research questions

• RQ1 Can selective knowledge transfer improve training efficiency without reducing model diversity in deep neural network ensembles?
• RQ2 Does a diversity-driven loss function that promotes negative correlation among base models lead to higher ensemble accuracy?
• RQ3 Can a Boosting-based framework further enhance diversity and performance when combined with selective knowledge transfer and diversity optimization?
• RQ4 How does the trade-off between accuracy and diversity, controlled by hyperparameter γ, affect the final ensemble performance?
• RQ5 Does EDDE outperform existing ensemble methods in terms of both accuracy and training cost under the same budget?

Key findings

• EDDE achieves the highest ensemble accuracy of 74.38% on CIFAR-100 with ResNet-32 under a 200-epoch training budget, outperforming all baselines.
• The optimal hyperparameter γ = 0.1 achieves the best balance between diversity and accuracy, yielding the highest ensemble performance.
• EDDE with γ = 0 (normal loss) achieves 73.86% accuracy, confirming that diversity-driven loss improves performance over standard training.
• EDDE with full knowledge transfer (EDDE (transfer all)) achieves 73.37% accuracy but lower diversity (0.1631), indicating trade-offs in transfer strategy.
• EDDE (transfer none) achieves the highest diversity (0.1854) but the lowest accuracy (70.78%), showing that knowledge transfer is essential for convergence under limited budget.
• EDDE outperforms AdaBoost.NC with transfer learning (72.64% accuracy), demonstrating superior balance between diversity and accuracy.