[论文解读] Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning

Sebastian Raschka|arXiv (Cornell University)|Nov 13, 2018

Machine Learning and Data Classification被引用 574

一句话总结

对模型评估、模型选择与算法选择技术的全面综述，包含优缺点、实用指导，以及对小数据集和多重比较的建议。

ABSTRACT

The correct use of model evaluation, model selection, and algorithm selection techniques is vital in academic machine learning research as well as in many industrial settings. This article reviews different techniques that can be used for each of these three subtasks and discusses the main advantages and disadvantages of each technique with references to theoretical and empirical studies. Further, recommendations are given to encourage best yet feasible practices in research and applications of machine learning. Common methods such as the holdout method for model evaluation and selection are covered, which are not recommended when working with small datasets. Different flavors of the bootstrap technique are introduced for estimating the uncertainty of performance estimates, as an alternative to confidence intervals via normal approximation if bootstrapping is computationally feasible. Common cross-validation techniques such as leave-one-out cross-validation and k-fold cross-validation are reviewed, the bias-variance trade-off for choosing k is discussed, and practical tips for the optimal choice of k are given based on empirical evidence. Different statistical tests for algorithm comparisons are presented, and strategies for dealing with multiple comparisons such as omnibus tests and multiple-comparison corrections are discussed. Finally, alternative methods for algorithm selection, such as the combined F-test 5x2 cross-validation and nested cross-validation, are recommended for comparing machine learning algorithms when datasets are small.

研究动机与目标

• 明确评估泛化性能、模型选择与算法选择的角色与目标。
• 综述常见方法（保留集、交叉验证、自助法）的偏差、权衡与适用性。
• 提供实用建议，以提高研究严谨性和实际应用中的机器学习实践。

提出的方法

• 解释保留集验证及其偏差，包括分层和悲观偏差。
• 介绍自助法及其如何估计性能不确定性。
• 讨论交叉验证（包括 k 折、留一法）以及超参数调优与模型选择之间的关系。
• 给出算法比较的统计检验方法（如 McNemar、F 检验、Cochrane’s Q）及多重检验校正。
• 描述用于算法比较的高级策略（如 5x2 交叉验证、嵌套交叉验证）。
• 强调小数据集与简约法则的考虑因素。

实验结果

研究问题

• RQ1 不同评估策略（保留集、交叉验证、自助法）如何估计泛化性能并支持模型/算法选择？
• RQ2 每种方法会产生哪些偏差与方差，分层与样本量如何影响它们？
• RQ3 在数据集大小受限的情况下，哪些检验与校正程序最适合比较多种算法？
• RQ4 在研究与应用中，哪些实际建议可优化模型评估的可靠性与有效性？

主要发现

• 保留法可能引入乐观偏差，且不太适用于小数据集；分层有助于缓解类别分布问题。
• 自助法和重复抽样提供估计性能不确定性的方法。
• k 折交叉验证具有受 k 控制的偏差-方差权衡；给出选择 k 及用于模型选择的指南。
• 用于比较分类器的统计检验（如 McNemar、F 检验、Cochrane’s Q）以及多重检验校正被讨论。
• 对于小型数据集，推荐使用如 5x2 交叉验证和嵌套交叉验证等组合方法来进行算法比较。
• 文章强调泛化性能与模型/算法选择中的相对性能之间的区别。