[论文解读] How Much Automation Does a Data Scientist Want?

Dakuo Wang, Q. Vera Liao+6 more|arXiv (Cornell University)|Jan 7, 2021

Persona Design and Applications参考文献 111被引用 25

一句话总结

该论文提出一个全面的人类在环 AutoML 框架，具有 5 个自动化级别、10 个生命周期阶段、43 个子任务、5 种解释类型和 6 个用户角色，然后对 217 位 DS/ML 工作者进行调查以研究期望的自动化和解释，主张反对端到端全自动化。

ABSTRACT

Data science and machine learning (DS/ML) are at the heart of the recent advancements of many Artificial Intelligence (AI) applications. There is an active research thread in AI, \autoai, that aims to develop systems for automating end-to-end the DS/ML Lifecycle. However, do DS and ML workers really want to automate their DS/ML workflow? To answer this question, we first synthesize a human-centered AutoML framework with 6 User Role/Personas, 10 Stages and 43 Sub-Tasks, 5 Levels of Automation, and 5 Types of Explanation, through reviewing research literature and marketing reports. Secondly, we use the framework to guide the design of an online survey study with 217 DS/ML workers who had varying degrees of experience, and different user roles "matching" to our 6 roles/personas. We found that different user personas participated in distinct stages of the lifecycle -- but not all stages. Their desired levels of automation and types of explanation for AutoML also varied significantly depending on the DS/ML stage and the user persona. Based on the survey results, we argue there is no rationale from user needs for complete automation of the end-to-end DS/ML lifecycle. We propose new next steps for user-controlled DS/ML automation.

研究动机与目标

• Synthesize a human-centered AutoML framework covering lifecycle stages, tasks, automation levels, and explanations.
• Investigate DS/ML practitioners' needs and preferences for automation and explanations through an online survey.
• Assess whether there is a demand for complete end-to-end automation of the DS/ML lifecycle.
• Provide guidance for future human-in-the-loop AutoML research and design.

提出的方法

• Review literature to construct a 6-role, 10-stage, 43-subtask AutoML framework with 5 levels of automation and 5 types of explanation.
• Define the DS/ML lifecycle framework and classify stages and sub-tasks.
• Design an online survey targeting 217 DS/ML workers aligned to the framework’s personas.
• Analyze survey responses to identify variation in preferred automation levels and explanations across stages and roles.
• Synthesize findings to argue against prioritizing full end-to-end automation and propose future HITL AutoML directions.

实验结果

研究问题

• RQ1 What automation levels do different DS/ML roles desire at various lifecycle stages?
• RQ2 What types of explanations do users want to accompany AutoML outputs across stages and roles?
• RQ3 Is there evidence supporting the need for end-to-end automation of the DS/ML lifecycle?
• RQ4 How should AutoML systems be designed to support human-in-the-loop collaboration across roles?
• RQ5 What are the implications for future research and design in human-centered AutoML?

主要发现

• Different personas participate in distinct lifecycle stages and tasks, not uniformly across the lifecycle.
• Desired automation levels and explanation types vary significantly by lifecycle stage and user role.
• There is no strong justification from user needs for complete (L4) end-to-end automation of the lifecycle.
• The framework aligns with practitioner experiences and supports user-centered design directions for AutoML systems.
• There is a need for next steps that emphasize user-controlled DS/ML automation rather than full automation.