[Paper Review] Diversity and Generalization in Neural Network Ensembles

Luis Ángel Ortega Cuesta, Rafael Cabañas+1 more|arXiv (Cornell University)|Oct 26, 2021

Adversarial Robustness in Machine Learning50 references4 citations

TL;DR

This paper presents a theoretically grounded framework linking diversity to generalization in neural network ensembles, showing how diversity reduces ensemble error across different loss functions (squared, cross-entropy, 0-1) and combination strategies (model averaging, weighted majority vote). It introduces a novel diversity measure, derives its impact on generalization error, and empirically validates that higher diversity correlates with better ensemble performance, especially when promoted via methods like random initialization or P2B-Ensemble.

ABSTRACT

Ensembles are widely used in machine learning and, usually, provide state-of-the-art performance in many prediction tasks. From the very beginning, the diversity of an ensemble has been identified as a key factor for the superior performance of these models. But the exact role that diversity plays in ensemble models is poorly understood, specially in the context of neural networks. In this work, we combine and expand previously published results in a theoretically sound framework that describes the relationship between diversity and ensemble performance for a wide range of ensemble methods. More precisely, we provide sound answers to the following questions: how to measure diversity, how diversity relates to the generalization error of an ensemble, and how diversity is promoted by neural network ensemble algorithms. This analysis covers three widely used loss functions, namely, the squared loss, the cross-entropy loss, and the 0-1 loss; and two widely used model combination strategies, namely, model averaging and weighted majority vote. We empirically validate this theoretical analysis with neural network ensembles.

Motivation & Objective

• To establish a theoretically sound relationship between diversity and generalization error in neural network ensembles.
• To define a general, loss-agnostic diversity measure applicable across different ensemble methods and loss functions.
• To analyze how existing neural network ensemble algorithms promote diversity and affect ensemble performance.
• To validate the theoretical framework empirically using diverse architectures and datasets.
• To clarify the conditions under which ensembles outperform individual models, based on diversity and individual accuracy.

Proposed method

• Derives a novel diversity measure, D(ρ), based on the correlation between individual models' predictions.
• Uses PAC-Bayesian bounds to decompose ensemble generalization error into individual error and diversity components.
• Applies the framework to three loss functions: squared error, cross-entropy, and 0-1 loss.
• Analyzes two model combination strategies: model averaging and weighted majority vote.
• Empirically evaluates the framework on LeNet5, ResNet20, and MLP50 across CIFAR-10, CIFAR-100, and WineQuality datasets.
• Uses ensemble size variation and algorithms like Ensemble and P2B-Ensemble to study diversity scaling and performance.

Experimental results

Research questions

• RQ1 How can diversity in neural network ensembles be formally measured in a way that connects to generalization error?
• RQ2 How does diversity influence the generalization error of an ensemble across different loss functions?
• RQ3 What is the trade-off between individual model performance, diversity, and ensemble generalization error?
• RQ4 How do common neural network ensemble algorithms (e.g., random initialization, P2B-Ensemble) promote diversity?
• RQ5 Under what conditions does an ensemble outperform the best single model, and how can this be predicted via diversity?

Key findings

• Ensembles with higher diversity, as measured by D(ρ), consistently show a larger performance gap between individual models and the ensemble, confirming the error-cancellation effect.
• Random initialization in simple networks like LeNet5 and MLP50 fails to generate high diversity, limiting ensemble gains, especially under squared loss.
• For deeper networks like ResNet20, increasing ensemble size steadily increases diversity, leading to improved generalization.
• The P2B-Ensemble method consistently increases diversity with ensemble size, outperforming standard ensemble training in low-diversity regimes.
• Corollary 5 provides a practical condition for ensemble superiority: when the diversity term outweighs individual error, the ensemble outperforms the best single model.
• Empirical results show that for cross-entropy and 0-1 loss, all ensembles trained with standard methods satisfy the condition in Corollary 5 and generalize better than the best individual model.