[Paper Review] Benchmarking Automatic Machine Learning Frameworks

Adithya Balaji, Alexander G. Allen|arXiv (Cornell University)|Aug 17, 2018

Cloud Computing and Resource Management2 references60 citations

TL;DR

This paper benchmarks open-source AutoML frameworks (auto-sklearn, TPOT, auto_ml, H2O AutoML) on OpenML datasets for classification and regression to compare performance patterns.

ABSTRACT

AutoML serves as the bridge between varying levels of expertise when designing machine learning systems and expedites the data science process. A wide range of techniques is taken to address this, however there does not exist an objective comparison of these techniques. We present a benchmark of current open source AutoML solutions using open source datasets. We test auto-sklearn, TPOT, auto_ml, and H2O's AutoML solution against a compiled set of regression and classification datasets sourced from OpenML and find that auto-sklearn performs the best across classification datasets and TPOT performs the best across regression datasets.

Motivation & Objective

• Assess the performance of leading open-source AutoML frameworks across a diverse set of datasets.
• Provide a fair, reproducible benchmarking methodology using OpenML data and standardized metrics.
• Identify strengths, weaknesses, and practical considerations for AutoML deployment in time-constrained settings.

Proposed method

• Benchmark a fixed set of 57 classification and 30 regression OpenML datasets.
• Standardize framework configurations to enable fair comparisons (time limits, seeds, metrics).
• Use weighted F1-score for classification and MSE (inverted for comparability) for regression as primary metrics.
• Employ a distributed computing setup (AWS Batch and bare-metal) to handle large compute demands.
• Document dataset preprocessing requirements and framework-specific limitations.
• Analyze pairwise framework performance and dataset-dependent trends.

Experimental results

Research questions

• RQ1 Which AutoML framework achieves the highest performance on classification tasks across OpenML datasets?
• RQ2 Which framework dominates on regression tasks?
• RQ3 How do framework performances vary with dataset characteristics such as size, dimensionality, and class distribution?
• RQ4 What practical limitations and failures arise when scaling AutoML frameworks in distributed environments?

Key findings

• Auto-sklearn performs best on classification datasets.
• TPOT performs best on regression datasets.
• Overall results exhibit high variance across datasets and seeds, highlighting codebase and feature set influences.
• AutoML frameworks show notable failures under certain memory and time constraints, necessitating careful resource management.
• Datasets and seeds influence results significantly, warranting robust benchmarking design.