[论文解读] Drawing Parallels between Multi-Label Classification and Multi-Target Regression

Eleftherios Spyromitros-Xioufis, William Groves+2 more|arXiv (Cornell University)|Nov 28, 2012

Text and Document Classification Technologies被引用 1

一句话总结

该论文将两种成功的多标签分类方法——堆叠单目标（Stacked Single-Target）与回归链集成（Ensemble of Regressor Chains）——适配于多目标回归任务，解决了目标变量估计中训练与预测阶段之间的关键差异问题。通过使用样本外估计缓解该问题后，所提出的方法在多种数据集上显著优于独立回归和当前最先进的多任务学习方法。

ABSTRACT

In many practical applications of supervised learning the task involves the prediction of multiple target variables from a common set of input variables. When the prediction targets are binary the task is called multi-label classification, while when the targets are continuous the task is called multi-target regression. In both tasks, target variables often exhibit statistical dependencies and exploiting them in order to improve predictive accuracy is a core challenge. A family of multi-label classification methods address this challenge by building a separate model for each target on an expanded input space where other targets are treated as additional input variables. Despite the success of these methods in the multi-label classification domain, their applicability and effectiveness in multi-target regression has not been studied until now. In this paper, we introduce two new methods for multi-target regression, called Stacked Single-Target and Ensemble of Regressor Chains, by adapting two popular multi-label classification methods of this family. Furthermore, we highlight an inherent problem of these methods - a discrepancy of the values of the additional input variables between training and prediction - and develop extensions that use out-of-sample estimates of the target variables during training in order to tackle this problem. The results of an extensive experimental evaluation carried out on a large and diverse collection of datasets show that, when the discrepancy is appropriately mitigated, the proposed methods attain consistent improvements over the independent regressions baseline. Moreover, two versions of Ensemble of Regression Chains perform significantly better than four state-of-the-art methods including regularization-based multi-task learning methods and a multi-objective random forest approach.

研究动机与目标

• 将成功的多标签分类技术扩展至多目标回归设置。
• 解决这些方法在训练与预测阶段目标变量值之间固有的差异问题。
• 开发能够利用多个连续目标之间统计依赖关系的改进回归模型。
• 评估所提方法相对于独立回归和当前最先进的多任务学习方法的有效性。

提出的方法

• 通过在扩展的输入空间中将其他目标作为特征，为每个目标单独训练回归器，将多标签分类中的堆叠单目标方法适配至多目标回归。
• 通过使用随机化目标顺序训练多个回归链，将回归链集成方法进行适配，以捕捉目标之间的相互依赖关系。
• 提出一种新型训练流程，在训练过程中使用目标变量的样本外估计，以解决训练与预测阶段之间的差异问题。
• 采用基于交叉验证的估计方法，生成用于训练过程中作为输入特征的目标变量的可靠样本外预测。
• 在回归链集成方法中，通过多个链的集成平均来提升鲁棒性与预测准确性。
• 在堆叠与集成框架中使用标准回归模型（如线性模型、随机森林）作为基学习器。

实验结果

研究问题

• RQ1 既有的多标签分类方法能否被有效适配于多目标回归任务？
• RQ2 训练与预测阶段目标值之间的差异如何影响这些适配方法的模型性能？
• RQ3 在训练过程中使用目标变量的样本外估计，能否缓解由此差异导致的性能下降？
• RQ4 所提方法是否在显著程度上优于独立回归和现有最先进的多任务学习技术？

主要发现

• 当正确处理训练-预测差异时，所提出的堆叠单目标与回归链集成方法在预测准确性上持续优于独立回归基线。
• 回归链集成方法的两种变体显著优于四种当前最先进的方法，包括基于正则化的多任务学习与多目标随机森林方法。
• 在训练过程中使用样本外估计能有效解决差异问题，且对实现性能提升至关重要。
• 在大量且多样化的数据集上均观察到性能提升，表明方法具有广泛的适用性与鲁棒性。
• 从多标签分类到多目标回归的方法论适配是有效且可迁移的，且得到了强有力的实证验证。
• 结果表明，通过结构化输入扩展建模目标间依赖关系，可在多目标回归中带来显著的性能提升。