[Paper Review] Ensemble Squared: A Meta AutoML System

Jason J. Yoo, Tony Joseph+3 more|arXiv (Cornell University)|Dec 10, 2020

Machine Learning and Data Classification20 references4 citations

TL;DR

Ensemble Squared (Ensemble 2) is a meta AutoML system that ensembles pipelines from multiple diverse open-source AutoML tools to achieve state-of-the-art performance on tabular classification benchmarks. By leveraging the complementary search strategies and model spaces of base AutoML systems, it improves accuracy and robustness, outperforming individual systems including AutoGluon in wall-clock time comparisons.

ABSTRACT

There are currently many barriers that prevent non-experts from exploiting machine learning solutions ranging from the lack of intuition on statistical learning techniques to the trickiness of hyperparameter tuning. Such barriers have led to an explosion of interest in automated machine learning (AutoML), whereby an off-the-shelf system can take care of many of the steps for end-users without the need for expertise in machine learning. This paper presents Ensemble Squared (Ensemble$^2$), an AutoML system that ensembles the results of state-of-the-art open-source AutoML systems. Ensemble$^2$ exploits the diversity of existing AutoML systems by leveraging the differences in their model search space and heuristics. Empirically, we show that diversity of each AutoML system is sufficient to justify ensembling at the AutoML system level. In demonstrating this, we also establish new state-of-the-art AutoML results on the OpenML tabular classification benchmark.

Motivation & Objective

• To address the barrier non-experts face in applying machine learning due to complexity in model selection and hyperparameter tuning.
• To investigate whether ensembling multiple AutoML systems can yield measurable performance gains due to their inherent diversity in search strategies and model spaces.
• To build a robust, accessible AutoML system that combines the strengths of multiple existing AutoML tools.
• To establish a new state-of-the-art baseline on the OpenML tabular classification benchmark using meta-ensembling of AutoML systems.

Proposed method

• The system uses a two-stage pipeline: first, it runs M base AutoML systems in parallel via Singularity containers to search for machine learning pipelines within a fixed time limit.
• Each base system performs its own pipeline search using its unique search space, heuristics, and hyperparameter optimization strategies, exploiting diversity in model and preprocessing choices.
• After the search phase, the top K pipelines (ranked by validation score) from all base systems are selected and optionally retrained on the full training set.
• The selected pipelines are then ensembled using either majority voting or a stacking meta-learner to produce final test set predictions.
• The system is designed to be robust: it only fails when all base AutoML systems fail, improving reliability over individual systems.
• A public web interface enables non-experts to upload datasets and receive predictions with minimal technical expertise.

Experimental results

Research questions

• RQ1 Can ensembling multiple AutoML systems lead to improved performance compared to individual systems?
• RQ2 Is there sufficient diversity in the performance of existing AutoML systems to justify meta-ensembling at the AutoML level?
• RQ3 How does Ensemble 2 compare to state-of-the-art AutoML systems under both wall-clock and equal-compute time constraints?
• RQ4 To what extent do the base AutoML systems exhibit correlated or uncorrelated performance across different datasets?

Key findings

• Ensemble 2 achieved a new state-of-the-art result on the OpenML tabular classification benchmark, outperforming all individual AutoML systems in wall-clock time comparisons.
• In a one-hour wall-clock comparison, Ensemble 2 achieved first place on 31 out of 41 OpenML datasets, while AutoGluon achieved first place on 30.
• Under equal-compute conditions (1-hour vs. 5-hour runs), Ensemble 2 remained competitive, with AutoGluon performing best overall, though other systems showed slight performance drops on longer runs.
• Correlation analysis revealed that base AutoML systems exhibit moderate to low correlation (e.g., r ≈ 0.79 between AutoGluon and CMU AutoML), indicating sufficient diversity to justify meta-ensembling.
• The system demonstrated improved robustness, failing only when all base AutoML systems failed, highlighting its resilience to individual system failures.
• The performance gain from ensembling is attributed to the diversity in search spaces, heuristics, and preprocessing strategies across base AutoML systems.