[Paper Review] When are ensembles really effective?

Ryan Theisen, Hyunsuk Kim+3 more|arXiv (Cornell University)|May 21, 2023

Machine Learning and Data Classification4 citations

TL;DR

This paper introduces the ensemble improvement rate (EIR) and disagreement-error ratio (DER) to theoretically and empirically determine when ensembling significantly improves classification performance. It proves that ensembling is most effective when DER > 1—i.e., when models disagree substantially relative to their error rate—and shows that interpolating models (e.g., deep neural networks) see far less benefit than non-interpolating models (e.g., random forests), where ensembling remains highly effective even beyond the interpolation threshold.

ABSTRACT

Ensembling has a long history in statistical data analysis, with many impactful applications. However, in many modern machine learning settings, the benefits of ensembling are less ubiquitous and less obvious. We study, both theoretically and empirically, the fundamental question of when ensembling yields significant performance improvements in classification tasks. Theoretically, we prove new results relating the \emph{ensemble improvement rate} (a measure of how much ensembling decreases the error rate versus a single model, on a relative scale) to the \emph{disagreement-error ratio}. We show that ensembling improves performance significantly whenever the disagreement rate is large relative to the average error rate; and that, conversely, one classifier is often enough whenever the disagreement rate is low relative to the average error rate. On the way to proving these results, we derive, under a mild condition called \emph{competence}, improved upper and lower bounds on the average test error rate of the majority vote classifier. To complement this theory, we study ensembling empirically in a variety of settings, verifying the predictions made by our theory, and identifying practical scenarios where ensembling does and does not result in large performance improvements. Perhaps most notably, we demonstrate a distinct difference in behavior between interpolating models (popular in current practice) and non-interpolating models (such as tree-based methods, where ensembling is popular), demonstrating that ensembling helps considerably more in the latter case than in the former.

Motivation & Objective

• To resolve the ambiguity in when ensembling provides non-trivial performance gains in modern machine learning.
• To formalize the conditions under which ensembling improves or fails to improve classification accuracy.
• To empirically compare ensembling effectiveness across interpolating (e.g., deep neural networks) and non-interpolating (e.g., tree-based) models.
• To introduce and validate the competence assumption as a minimal, realistic condition for theoretical guarantees on ensemble performance.
• To provide practical guidelines for when practitioners should invest in training and deploying ensembles.

Proposed method

• Define the ensemble improvement rate (EIR) as a relative measure of error reduction from ensembling versus a single model.
• Introduce the disagreement-error ratio (DER) as a key predictor of ensemble effectiveness, measuring disagreement relative to average error.
• Propose the 'competence' assumption to exclude pathological cases and enable tighter theoretical bounds on EIR.
• Prove that under competence, EIR is bounded by linear functions of DER, with EIR increasing when DER > 1.
• Use empirical validation across diverse tasks, including deep ensembles, random feature models, and fine-tuned BERT, to verify theoretical predictions.
• Compare ensembling behavior across model classes, particularly distinguishing interpolating (zero training error) and non-interpolating models.

Experimental results

Research questions

• RQ1 Under what conditions does ensembling produce significant performance improvements in classification tasks?
• RQ2 How does the disagreement-error ratio (DER) predict the magnitude of ensemble improvement?
• RQ3 Why do interpolating models (e.g., deep neural networks) show limited gains from ensembling compared to non-interpolating models (e.g., random forests)?
• RQ4 To what extent does the competence assumption hold in real-world machine learning settings?
• RQ5 Can the linear relationship between EIR and DER be empirically verified across diverse model architectures and datasets?

Key findings

• Ensembling significantly improves performance when the disagreement-error ratio (DER) exceeds 1, indicating high disagreement relative to error rate.
• The ensemble improvement rate (EIR) is upper and lower bounded by linear functions of DER under the competence assumption, providing a theoretical foundation for predicting ensemble gains.
• Interpolating ensembles (e.g., deep neural networks, random feature models) exhibit consistently low EIR, with improvement rates dropping sharply after the interpolation threshold.
• Non-interpolating models like random forests maintain high EIR and DER even as model capacity increases, and both metrics plateau only after reaching zero training error.
• Empirical results confirm that EIR correlates strongly with DER across diverse settings, validating the theoretical bounds.
• Fine-tuned BERT ensembles on GLUE tasks show uniformly low DER (<1) and low EIR, consistent with the theory that interpolating models gain little from ensembling.