[Paper Review] DeepCAVE: An Interactive Analysis Tool for Automated Machine Learning

René Sass, Eddie Bergman+3 more|arXiv (Cornell University)|Jun 7, 2022

Machine Learning and Data Classification4 citations

TL;DR

DeepCAVE is an interactive, real-time analysis framework for Automated Machine Learning (AutoML) that enhances transparency and trust by enabling users to monitor, visualize, and interpret hyperparameter optimization, neural architecture search, and multi-fidelity/multi-objective runs while they are still executing. It supports dynamic exploration through modular plugins for performance analysis, configuration importance, and Pareto front visualization, with key contributions including real-time monitoring, multi-fidelity optimization support, and user-friendly interpretation tools for both experts and non-experts.

ABSTRACT

Automated Machine Learning (AutoML) is used more than ever before to support users in determining efficient hyperparameters, neural architectures, or even full machine learning pipelines. However, users tend to mistrust the optimization process and its results due to a lack of transparency, making manual tuning still widespread. We introduce DeepCAVE, an interactive framework to analyze and monitor state-of-the-art optimization procedures for AutoML easily and ad hoc. By aiming for full and accessible transparency, DeepCAVE builds a bridge between users and AutoML and contributes to establishing trust. Our framework's modular and easy-to-extend nature provides users with automatically generated text, tables, and graphic visualizations. We show the value of DeepCAVE in an exemplary use-case of outlier detection, in which our framework makes it easy to identify problems, compare multiple runs and interpret optimization processes. The package is freely available on GitHub https://github.com/automl/DeepCAVE.

Motivation & Objective

• To address the lack of transparency and trust in AutoML systems due to their black-box nature.
• To enable real-time, interactive analysis of AutoML optimization processes during execution, not just post-hoc.
• To support multi-fidelity and multi-objective optimization workflows, which are common in deep learning AutoML.
• To provide accessible, intuitive visualizations and interpretability tools for users of all experience levels.
• To bridge the gap between users and AutoML by bringing the human back into the loop through interactive, modular analysis plugins.

Proposed method

• DeepCAVE is built as a modular, extensible framework that integrates with any AutoML optimizer via file system-based data converters.
• It supports real-time monitoring by continuously reading and parsing optimizer outputs as they are written during execution.
• The framework includes interactive plugins for visualizing performance trajectories, Pareto fronts, hyperparameter importance (via fANOVA and LPI), and configuration footprints.
• It enables comparison across multiple optimization runs or groups using shared visualizations and interactive filtering.
• The system generates auto-generated code snippets and configuration details on hover or click, allowing direct reuse of top-performing configurations.
• It is designed to be optimizer-agnostic, supporting a wide range of AutoML workflows including HPO, NAS, and CASH.

Experimental results

Research questions

• RQ1 How can users gain real-time insights into the progress and behavior of AutoML optimization processes without waiting for completion?
• RQ2 What are the most influential hyperparameters and architectural choices in a given AutoML run, and how do they affect performance?
• RQ3 How can multi-fidelity and multi-objective optimization results be effectively visualized and compared in an interactive manner?
• RQ4 Can interactive analysis tools improve user trust and understanding of AutoML decisions, especially in safety-critical applications?
• RQ5 How can interpretability plugins help users detect issues, identify outliers, and make informed decisions during long-running optimization processes?

Key findings

• DeepCAVE enables real-time monitoring of AutoML optimization processes, allowing users to detect issues early and make informed decisions during execution.
• The framework successfully supports multi-fidelity and multi-objective optimization, which are critical for efficient and realistic AutoDL workflows.
• Using fANOVA and LPI plugins, the study identified that learning rate, batch size, and model architecture were the most influential hyperparameters in the outlier detection task.
• The Pareto front visualization allowed users to compare trade-offs between objectives such as accuracy, training time, and cost, supporting informed configuration selection.
• Interactive features like clickable points and auto-generated code snippets significantly improved usability and reusability of top-performing configurations.
• The framework outperforms existing tools like CAVE and XAutoML by offering real-time, interactive, and extensible analysis with support for complex optimization workflows.