[论文解读] A Multivocal Literature Review on the Benefits and Limitations of Automated Machine Learning Tools

Kelly Azevedo, Luigi Quaranta+2 more|arXiv (Cornell University)|Jan 21, 2024

Machine Learning and Data Classification被引用 7

一句话总结

这篇论文进行一次多元声音的文献综述，综合了来自54个学术来源和108份灰色文献的AutoML工具的利益与局限，强调对ML工作流程和从业者可及性的影响。

ABSTRACT

Context. Advancements in Machine Learning (ML) are revolutionizing every application domain, driving unprecedented transformations and fostering innovation. However, despite these advances, several organizations are experiencing friction in the adoption of ML-based technologies, mainly due to the shortage of ML professionals. In this context, Automated Machine Learning (AutoML) techniques have been presented as a promising solution to democratize ML adoption. Objective. We aim to provide an overview of the evidence on the benefits and limitations of using AutoML tools. Method. We conducted a multivocal literature review, which allowed us to identify 54 sources from the academic literature and 108 sources from the grey literature reporting on AutoML benefits and limitations. We extracted reported benefits and limitations from the papers and applied thematic analysis. Results. We identified 18 benefits and 25 limitations. Concerning the benefits, we highlight that AutoML tools can help streamline the core steps of ML workflows, namely data preparation, feature engineering, model construction, and hyperparameter tuning, with concrete benefits on model performance, efficiency, and scalability. In addition, AutoML empowers both novice and experienced data scientists, promoting ML accessibility. On the other hand, we highlight several limitations that may represent obstacles to the widespread adoption of AutoML. For instance, AutoML tools may introduce barriers to transparency and interoperability, exhibit limited flexibility for complex scenarios, and offer inconsistent coverage of the ML workflow. Conclusions. The effectiveness of AutoML in facilitating the adoption of machine learning by users may vary depending on the tool and the context in which it is used. As of today, AutoML tools are used to increase human expertise rather than replace it, and, as such, they require skilled users.

研究动机与目标

• 识别并对文献中报道的AutoML工具的利益进行分类。
• 识别并对AutoML adoption报道的局限性和障碍进行分类。
• 评估AutoML如何影响数据准备、特征工程、模型构建和超参数调优。
• 评估谁受益于AutoML，以及在何种情境下它能提供支持或存在不足。

提出的方法

• 进行一次多元声音的文献综述，以捕捉学术与灰色文献的观点。
• 应用主题分析来提取利益与局限。
• 将发现汇总成一个带有支持来源的结构化的利益与局限清单。

实验结果

研究问题

• RQ1 AutoML工具为ML工作流程（数据准备、特征工程、模型构建、超参数调优）提供了哪些收益？
• RQ2 有哪些局限性阻碍AutoML工具的广泛采用，在何种情境下这些工具表现不足或无法满足需求？
• RQ3 谁从AutoML中受益，工具特征如何影响新手与有经验的数据科学家的可访问性？
• RQ4 AutoML工具如何影响ML项目中的透明度和互操作性？

主要发现

• AutoML工具可以简化核心ML工作流程步骤，提升模型性能、效率和可扩展性。
• AutoML让初学者和有经验的数据科学家都更有能力，促进ML的可访问性。
• 局限性包括对透明度和互操作性的障碍、对复杂情境的灵活性有限，以及对ML工作流程覆盖的不一致。
• AutoML的有效性因工具和情境而异，这些工具目前是增加而非替代人类专业知识。