[Paper Review] A Survey on Ensemble Learning under the Era of Deep Learning

Yongquan Yang, Haijun Lv+1 more|arXiv (Cornell University)|Jan 21, 2021

Anomaly Detection Techniques and Applications162 references4 citations

TL;DR

This survey provides a comprehensive overview of ensemble learning in the deep learning era, analyzing methodologies, recent advances, and challenges in deploying ensemble deep learning due to high computational costs. It identifies key technical hurdles and proposes directions for efficient, scalable ensemble methods to enhance model generalization while reducing resource demands in real-world applications.

ABSTRACT

Due to the dominant position of deep learning (mostly deep neural networks) in various artificial intelligence applications, recently, ensemble learning based on deep neural networks (ensemble deep learning) has shown significant performances in improving the generalization of learning system. However, since modern deep neural networks usually have millions to billions of parameters, the time and space overheads for training multiple base deep learners and testing with the ensemble deep learner are far greater than that of traditional ensemble learning. Though several algorithms of fast ensemble deep learning have been proposed to promote the deployment of ensemble deep learning in some applications, further advances still need to be made for many applications in specific fields, where the developing time and computing resources are usually restricted or the data to be processed is of large dimensionality. An urgent problem needs to be solved is how to take the significant advantages of ensemble deep learning while reduce the required expenses so that many more applications in specific fields can benefit from it. For the alleviation of this problem, it is essential to know about how ensemble learning has developed under the era of deep learning. Thus, in this article, we present fundamental discussions focusing on data analyses of published works, methodologies, recent advances and unattainability of traditional ensemble learning and ensemble deep learning. We hope this article will be helpful to realize the intrinsic problems and technical challenges faced by future developments of ensemble learning under the era of deep learning.

Motivation & Objective

• To analyze the evolution and current state of ensemble learning in the context of deep neural networks.
• To identify computational and resource limitations hindering widespread adoption of ensemble deep learning in real-world applications.
• To examine the trade-offs between performance gains and increased training/inference costs in ensemble deep learning.
• To highlight recent advances in fast ensemble learning techniques aimed at reducing time and space overhead.
• To provide a roadmap for future research addressing scalability and efficiency in ensemble deep learning.

Proposed method

• Systematic review and analysis of published works on ensemble learning with deep neural networks.
• Categorization of ensemble methods based on training strategies, model diversity, and aggregation techniques.
• Evaluation of computational costs and efficiency trade-offs across different ensemble architectures.
• Identification of key bottlenecks in training and inference for large-scale ensemble deep learning systems.
• Survey of recent fast ensemble learning algorithms designed to reduce computational overhead.
• Synthesis of open challenges and research directions for scalable, efficient ensemble deep learning.

Experimental results

Research questions

• RQ1 How has ensemble learning evolved under the rise of deep neural networks?
• RQ2 What are the primary computational and resource constraints limiting the deployment of ensemble deep learning in practice?
• RQ3 What techniques have been proposed to accelerate ensemble training and inference while maintaining performance?
• RQ4 How do modern ensemble methods balance model diversity, accuracy, and computational efficiency?
• RQ5 What are the unresolved challenges and future research directions in scalable ensemble deep learning?

Key findings

• Ensemble deep learning significantly improves model generalization but incurs substantial time and space overhead due to millions to billions of parameters in deep networks.
• Traditional ensemble learning methods are often infeasible for large-scale deep learning applications due to high computational demands.
• Recent fast ensemble learning algorithms have been proposed to reduce training and inference costs, but further improvements are needed for resource-constrained environments.
• The survey identifies a critical gap between the performance benefits of ensemble deep learning and its practical deployment due to resource limitations.
• Model diversity, training efficiency, and inference speed are key factors influencing the feasibility of ensemble deep learning in specific application domains.
• The authors conclude that future research must focus on developing lightweight, scalable, and efficient ensemble frameworks for real-world deployment.