[论文解读] A Survey on Ensemble Learning under the Era of Deep Learning

Yongquan Yang, Haijun Lv+1 more|arXiv (Cornell University)|Jan 21, 2021

Anomaly Detection Techniques and Applications参考文献 162被引用 4

一句话总结

本综述全面概述了深度学习时代集成学习的进展，分析了方法论、近期技术进展以及由于计算成本高昂而带来的部署集成深度学习的挑战。它识别出关键技术障碍，并提出了高效、可扩展集成方法的研究方向，以在降低实际应用中资源需求的同时提升模型泛化能力。

ABSTRACT

Due to the dominant position of deep learning (mostly deep neural networks) in various artificial intelligence applications, recently, ensemble learning based on deep neural networks (ensemble deep learning) has shown significant performances in improving the generalization of learning system. However, since modern deep neural networks usually have millions to billions of parameters, the time and space overheads for training multiple base deep learners and testing with the ensemble deep learner are far greater than that of traditional ensemble learning. Though several algorithms of fast ensemble deep learning have been proposed to promote the deployment of ensemble deep learning in some applications, further advances still need to be made for many applications in specific fields, where the developing time and computing resources are usually restricted or the data to be processed is of large dimensionality. An urgent problem needs to be solved is how to take the significant advantages of ensemble deep learning while reduce the required expenses so that many more applications in specific fields can benefit from it. For the alleviation of this problem, it is essential to know about how ensemble learning has developed under the era of deep learning. Thus, in this article, we present fundamental discussions focusing on data analyses of published works, methodologies, recent advances and unattainability of traditional ensemble learning and ensemble deep learning. We hope this article will be helpful to realize the intrinsic problems and technical challenges faced by future developments of ensemble learning under the era of deep learning.

研究动机与目标

• 分析深度神经网络背景下集成学习的演进过程与当前状态。
• 识别阻碍集成深度学习在实际应用中广泛采用的计算与资源限制。
• 研究集成深度学习中性能提升与训练/推理成本增加之间的权衡。
• 突出展示旨在降低时间和空间开销的快速集成学习技术的最新进展。
• 为解决集成深度学习中的可扩展性与效率问题，提供未来研究的路线图。

提出的方法

• 系统性回顾与分析关于深度神经网络集成学习的已发表文献。
• 基于训练策略、模型多样性与聚合技术对集成方法进行分类。
• 评估不同集成架构之间的计算成本与效率权衡。
• 识别大规模集成深度学习系统在训练与推理中的关键瓶颈。
• 调查近期旨在降低计算开销的快速集成学习算法。
• 综合总结可扩展、高效集成深度学习的开放挑战与研究方向。

实验结果

研究问题

• RQ1 在深度神经网络兴起的背景下，集成学习如何演变？
• RQ2 限制集成深度学习在实践中部署的主要计算与资源约束是什么？
• RQ3 提出了哪些技术以在保持性能的同时加速集成训练与推理？
• RQ4 现代集成方法如何在模型多样性、准确率与计算效率之间取得平衡？
• RQ5 可扩展集成深度学习中尚未解决的挑战与未来研究方向是什么？

主要发现

• 集成深度学习显著提升了模型泛化能力，但由于深度网络中包含数百万至数十亿参数，导致时间和空间开销巨大。
• 由于计算需求过高，传统集成学习方法在大规模深度学习应用中往往不可行。
• 近期已提出快速集成学习算法以降低训练与推理成本，但在资源受限环境中仍需进一步改进。
• 本综述识别出集成深度学习性能优势与其实际部署之间存在关键差距，主要受限于资源。
• 模型多样性、训练效率与推理速度是影响特定应用场景下集成深度学习可行性的关键因素。
• 作者得出结论：未来研究必须聚焦于开发轻量化、可扩展且高效的集成框架，以实现真实环境中的部署。