[Paper Review] How Much Automation Does a Data Scientist Want?

Dakuo Wang, Q. Vera Liao+6 more|arXiv (Cornell University)|Jan 7, 2021

Persona Design and Applications111 references25 citations

TL;DR

The paper proposes a comprehensive human-in-the-loop AutoML framework with 5 levels of automation, 10 lifecycle stages, 43 sub-tasks, 5 explanation types, and 6 user roles, then surveys 217 DS/ML workers to study desired automation and explanations, arguing against end-to-end full automation.

ABSTRACT

Data science and machine learning (DS/ML) are at the heart of the recent advancements of many Artificial Intelligence (AI) applications. There is an active research thread in AI, \autoai, that aims to develop systems for automating end-to-end the DS/ML Lifecycle. However, do DS and ML workers really want to automate their DS/ML workflow? To answer this question, we first synthesize a human-centered AutoML framework with 6 User Role/Personas, 10 Stages and 43 Sub-Tasks, 5 Levels of Automation, and 5 Types of Explanation, through reviewing research literature and marketing reports. Secondly, we use the framework to guide the design of an online survey study with 217 DS/ML workers who had varying degrees of experience, and different user roles "matching" to our 6 roles/personas. We found that different user personas participated in distinct stages of the lifecycle -- but not all stages. Their desired levels of automation and types of explanation for AutoML also varied significantly depending on the DS/ML stage and the user persona. Based on the survey results, we argue there is no rationale from user needs for complete automation of the end-to-end DS/ML lifecycle. We propose new next steps for user-controlled DS/ML automation.

Motivation & Objective

• Synthesize a human-centered AutoML framework covering lifecycle stages, tasks, automation levels, and explanations.
• Investigate DS/ML practitioners' needs and preferences for automation and explanations through an online survey.
• Assess whether there is a demand for complete end-to-end automation of the DS/ML lifecycle.
• Provide guidance for future human-in-the-loop AutoML research and design.

Proposed method

• Review literature to construct a 6-role, 10-stage, 43-subtask AutoML framework with 5 levels of automation and 5 types of explanation.
• Define the DS/ML lifecycle framework and classify stages and sub-tasks.
• Design an online survey targeting 217 DS/ML workers aligned to the framework’s personas.
• Analyze survey responses to identify variation in preferred automation levels and explanations across stages and roles.
• Synthesize findings to argue against prioritizing full end-to-end automation and propose future HITL AutoML directions.

Experimental results

Research questions

• RQ1 What automation levels do different DS/ML roles desire at various lifecycle stages?
• RQ2 What types of explanations do users want to accompany AutoML outputs across stages and roles?
• RQ3 Is there evidence supporting the need for end-to-end automation of the DS/ML lifecycle?
• RQ4 How should AutoML systems be designed to support human-in-the-loop collaboration across roles?
• RQ5 What are the implications for future research and design in human-centered AutoML?

Key findings

• Different personas participate in distinct lifecycle stages and tasks, not uniformly across the lifecycle.
• Desired automation levels and explanation types vary significantly by lifecycle stage and user role.
• There is no strong justification from user needs for complete (L4) end-to-end automation of the lifecycle.
• The framework aligns with practitioner experiences and supports user-centered design directions for AutoML systems.
• There is a need for next steps that emphasize user-controlled DS/ML automation rather than full automation.