[Paper Review] A Neophyte With AutoML: Evaluating the Promises of Automatic Machine Learning Tools

Oleg Bezrukavnikov, Rhema Linder|arXiv (Cornell University)|Jan 14, 2021

Machine Learning and Data Classification32 references4 citations

TL;DR

This paper evaluates three AutoML tools—TPOP, AutoKeras, and AutoGluon—from the perspective of a machine learning neophyte, assessing their usability, performance, and user experience on a banking dataset. Despite automation promises, the study reveals significant UX shortcomings, including confusing logs, poor training time estimation, and limited visualization of model internals, highlighting the need for more intuitive, visual interfaces to truly democratize AutoML for non-experts.

ABSTRACT

This paper discusses modern Auto Machine Learning (AutoML) tools from the perspective of a person with little prior experience in Machine Learning (ML). There are many AutoML tools both ready-to-use and under development, which are created to simplify and democratize usage of ML technologies in everyday life. Our position is that ML should be easy to use and available to a greater number of people. Prior research has identified the need for intuitive AutoML tools. This work seeks to understand how well AutoML tools have achieved that goal in practice. We evaluate three AutoML Tools to evaluate the end-user experience and system performance. We evaluate the tools by having them create models from a competition dataset on banking data. We report on their performance and the details of our experience. This process provides a unique understanding of the state of the art of AutoML tools. Finally, we use these experiences to inform a discussion on how future AutoML tools can improve the user experience for neophytes of Machine Learning.

Motivation & Objective

• To assess how well AutoML tools deliver on their promise of simplifying machine learning for non-experts.
• To evaluate the user experience (UX) of AutoML tools in real-world usage by a novice user.
• To identify gaps in AutoML tools related to logging, training time estimation, and visualization of model training processes.
• To provide actionable recommendations for improving AutoML tools to better serve beginners in machine learning.
• To explore the role of visual interfaces in making AutoML more accessible and democratized for general users.

Proposed method

• The study evaluates three widely used AutoML tools: TPOP, AutoKeras, and AutoGluon.
• A standardized banking dataset was used as the benchmark for model creation and performance comparison.
• User experience was assessed through hands-on experimentation, focusing on ease of setup, error handling, logging clarity, and visualization capabilities.
• Training processes were monitored using external tools like Weights & Biases and TensorDash to track metrics across multiple trials.
• The authors documented challenges such as log verbosity, lack of training time estimates, and overwriting of visualization data.
• The analysis included qualitative feedback on tool behavior, error messages, and the availability of visual dashboards for model monitoring.

Experimental results

Research questions

• RQ1 To what extent do AutoML tools successfully automate the machine learning pipeline for a user with no prior ML experience?
• RQ2 How do AutoML tools perform in terms of model accuracy and training efficiency on a real-world tabular dataset?
• RQ3 What are the key usability challenges faced by neophytes when using AutoML tools, particularly regarding logging, error messages, and training monitoring?
• RQ4 How do visualization and monitoring tools (e.g., Weights & Biases, TensorDash) enhance or fail to enhance the user experience during AutoML training?
• RQ5 What role can visual, no-code interfaces play in improving accessibility and democratizing machine learning for non-developers?

Key findings

• AutoML tools significantly reduce the need for manual hyperparameter tuning and model selection, but still require substantial user expertise in Python and library-specific syntax.
• Confusing and verbose log outputs hinder the user experience, especially for beginners, making it difficult to diagnose issues during training.
• Most tools lack reliable estimates for training duration, leading to uncertainty and potential user frustration during long-running experiments.
• Visualization of training progress and internal model states is limited; tools like TensorDash overwrite trial data, while Weights & Biases offer better multi-trial tracking.
• External monitoring tools such as Weights & Biases provide superior real-time visualization and comparison capabilities, but integration is not seamless and requires additional setup.
• The Northstar project exemplifies a promising future direction: visual, interactive, no-code interfaces that could greatly enhance accessibility and democratize AutoML for non-programmers.