[论文解读] An Easy to Use Repository for Comparing and Improving Machine Learning Algorithm Usage

Michael R. Smith, Andrew Dickson White+2 more|arXiv (Cornell University)|May 28, 2014

Machine Learning and Data Classification参考文献 15被引用 9

一句话总结

本文介绍了机器学习结果仓库（MLRR），这是一个轻量级、易于访问的数据库，用于存储机器学习实验的实例级预测结果。通过保留完整的实验细节——包括算法、超参数、训练集以及每个实例的预测结果——MLRR 实现了可复现的元学习、算法比较以及更优的模型选择，且设置开销极低。

ABSTRACT

The results from most machine learning experiments are used for a specific purpose and then discarded. This results in a significant loss of information and requires rerunning experiments to compare learning algorithms. This also requires implementation of another algorithm for comparison, that may not always be correctly implemented. By storing the results from previous experiments, machine learning algorithms can be compared easily and the knowledge gained from them can be used to improve their performance. The purpose of this work is to provide easy access to previous experimental results for learning and comparison. These stored results are comprehensive -- storing the prediction for each test instance as well as the learning algorithm, hyperparameters, and training set that were used. Previous results are particularly important for meta-learning, which, in a broad sense, is the process of learning from previous machine learning results such that the learning process is improved. While other experiment databases do exist, one of our focuses is on easy access to the data. We provide meta-learning data sets that are ready to be downloaded for meta-learning experiments. In addition, queries to the underlying database can be made if specific information is desired. We also differ from previous experiment databases in that our databases is designed at the instance level, where an instance is an example in a data set. We store the predictions of a learning algorithm trained on a specific training set for each instance in the test set. Data set level information can then be obtained by aggregating the results from the instances. The instance level information can be used for many tasks such as determining the diversity of a classifier or algorithmically determining the optimal subset of training instances for a learning algorithm.

研究动机与目标

• 解决通常在使用后即被丢弃的、难以访问且不可重用的机器学习实验结果的问题。
• 使研究人员无需重新运行实验即可比较学习算法和超参数。
• 通过提供全面的、实例级的数据，支持元学习，用于训练和评估元学习器。
• 通过提供即下即用的元数据集，降低元学习的入门门槛，减少对数据库模式或配置知识的需求。
• 通过存储每个实例的预测结果，支持分类器多样性分析、课程学习和实例难度表征等高级任务。

提出的方法

• 以实例级别存储机器学习实验结果，包括预测结果、学习算法、超参数以及所用训练集。
• 聚合实例级数据，推导出数据集级别的元特征和性能指标。
• 提供可下载的元数据集，用于元学习任务，如预测最优算法或超参数。
• 使用地标器（例如决策桩、平均节点学习器）高效计算具有代表性的元特征，用于实例级分析。
• 集成实例级难度度量，如 k 个不一致邻居、合取集大小、树深度、类别概率、少数类值和类别平衡度。
• 设计该仓库以实现易用性，避免严格的数据库模式或 PMML 依赖，实现无需学习曲线的直接使用。

实验结果

研究问题

• RQ1 如何以一种便于重用的方式存储和共享机器学习实验结果，以支持元学习？
• RQ2 实例级预测在多大程度上能提升元学习模型的准确性和可解释性？
• RQ3 轻量级、易访问的数据库在在多大程度上能减少比较学习算法和超参数所需的工作量？
• RQ4 实例级难度度量与分类难度和算法性能之间存在何种相关性？
• RQ5 存储完整的实验溯源信息（包括算法、超参数和训练集）对可复现性和基准测试有何影响？

主要发现

• MLRR 提供即下即用的元数据集，包含每个实例的预测结果、超参数、算法和训练集，可直接用于元学习实验。
• 实例级预测支持高级分析，例如通过分类器输出差异（COD）度量来衡量分类器多样性，并识别难以分类的实例。
• 难度度量如 k 个不一致邻居、合取集大小和类别概率能有效捕捉实例级难度，并与误分类率显著相关。
• 使用地标器可显著降低计算成本，同时保持具有竞争力的性能，实现元特征的高效计算。
• MLRR 通过识别基于预测模式的较易实例，支持课程学习，从而支持渐进式训练策略。
• 该仓库的轻量化设计相比传统数据库或基于 PMML 的系统显著降低了学习曲线，提升了可访问性和可扩展性。