[论文解读] Dense Adaptive Cascade Forest: A Densely Connected Deep Ensemble for Classification Problems

Haiyang Wang|arXiv (Cornell University)|Jan 1, 2018

Domain Adaptation and Few-Shot Learning参考文献 35被引用 1

一句话总结

本文提出Dense Adaptive Cascade Forest（daForest），一种通过层间密集残差连接、自适应超参数优化以及SAMME.R提升的深度集成模型，显著提升了分类准确率。该模型在无需预处理的高维稀疏数据上表现优异，超越传统模型与神经网络，达到当前最优性能。

ABSTRACT

Recent research has shown that deep ensemble for forest can achieve a huge increase in classification accuracy compared with the general ensemble learning method. Especially when there are only few training data. In this paper, we decide to take full advantage of this observation and introduce the Dense Adaptive Cascade Forest (daForest), which has better performance than the original one named Cascade Forest. And it is particularly noteworthy that daForest has a powerful ability to handle high-dimensional sparse data without any preprocessing on raw data like PCA or any other dimensional reduction methods. Our model is distinguished by three major features: the first feature is the combination of the SAMME.R boosting algorithm in the model, boosting gives the model the ability to continuously improve as the number of layer increases, which is not possible in stacking model or plain cascade forest. The second feature is our model connects each layer to its subsequent layers in a feed-forward fashion, to some extent this structure enhances the ability of the model to resist degeneration. When number of layers goes up, accuracy of model goes up a little in the first few layers then drop down quickly, we call this phenomenon degeneration in training stacking model. The third feature is that we add a hyper-parameter optimization layer before the first classification layer in the proposed deep model, which can search for the optimal hyper-parameter and set up the model in a brief period and nearly halve the training time without having too much impact on the final performance. Experimental results show that daForest performs particularly well on both high-dimensional low-order features and low-dimensional high-order features, and in some cases, even better than neural networks and achieves state-of-the-art results.

研究动机与目标

• 解决深度集成模型在训练过程中，特别是在堆叠和级联森林架构中出现的性能退化问题。
• 在无需PCA等数据预处理的前提下，提升高维稀疏特征与低维高阶特征的分类准确率。
• 通过引入自适应超参数优化层，加速模型配置，显著缩短训练时间，同时对最终性能影响极小。
• 通过在各层之间引入密集跳跃连接，提升模型稳定性与可扩展性，防止深层架构中准确率下降。

提出的方法

• 将SAMME.R提升算法集成到深度级联森林结构中，实现随着层数增加而持续提升准确率。
• 采用前馈密集连接机制，将每一层与所有后续层相连，改善梯度流动，减少深层模型中的退化现象。
• 在首个分类层之前引入超参数优化层，自动调节模型参数，将训练时间减少近一半。
• 直接以原始高维稀疏数据作为输入，不进行降维处理，充分利用模型对稀疏特征的内在鲁棒性。
• 采用级联架构，每一层对前序层的预测结果进行优化，同时通过提升动态调整样本权重，增强弱学习器性能。
• 引入类似残差的结构以稳定训练过程，并在模型深度增加时仍能保持性能增益。

实验结果

研究问题

• RQ1 深度集成森林模型能否在层数增加时维持或提升准确率，避免标准堆叠模型中常见的性能退化？
• RQ2 与传统集成方法相比，SAMME.R提升算法在深度级联森林架构中的集成对性能有何影响？
• RQ3 超参数优化层在不降低最终分类准确率的前提下，最多能将训练时间减少多少？
• RQ4 该模型能否在无需PCA或特征选择等预处理的情况下，实现在高维稀疏数据上的最先进性能？
• RQ5 密集残差连接机制如何提升深层森林模型的稳定性和泛化能力？

主要发现

• daForest在多个基准数据集上达到最先进性能，尤其在高维稀疏特征设置下表现卓越。
• 随着模型深度增加，该模型能持续保持准确率增益，避免了堆叠或普通级联森林中常见的性能急剧下降。
• 超参数优化层将训练时间减少近50%，同时最终模型准确率仅在可忽略范围内下降。
• 在某些数据集上，daForest优于传统随机森林集成模型与深度神经网络，尤其在稀疏高维输入场景下。
• 密集连接机制显著提升了模型稳定性，有效防止了深层架构中的性能退化。
• 该模型在低维高阶特征集上也表现出强大的泛化能力，表明其在多种数据类型中均具有广泛适用性。