[Paper Review] Efficient AutoML Pipeline Search with Matrix and Tensor Factorization

Chengrun Yang, Jicong Fan+2 more|arXiv (Cornell University)|Jun 7, 2020

Tensor decomposition and applications62 references4 citations

TL;DR

This paper proposes an efficient AutoML pipeline search system using matrix and tensor factorization to model pipeline performance across datasets. By leveraging low-rank surrogate models and greedy experiment design, it reduces the number of expensive pipeline evaluations needed, achieving state-of-the-art performance with under 10 seconds of design time on large-scale datasets.

ABSTRACT

Data scientists seeking a good supervised learning model on a new dataset have many choices to make: they must preprocess the data, select features, possibly reduce the dimension, select an estimation algorithm, and choose hyperparameters for each of these pipeline components. With new pipeline components comes a combinatorial explosion in the number of choices! In this work, we design a new AutoML system to address this challenge: an automated system to design a supervised learning pipeline. Our system uses matrix and tensor factorization as surrogate models to model the combinatorial pipeline search space. Under these models, we develop greedy experiment design protocols to efficiently gather information about a new dataset. Experiments on large corpora of real-world classification problems demonstrate the effectiveness of our approach.

Motivation & Objective

• Address the combinatorial explosion in AutoML pipeline search due to the vast number of component and hyperparameter combinations.
• Overcome the cold-start problem in AutoML by efficiently selecting informative pipelines to evaluate on new datasets.
• Develop surrogate models that generalize across datasets using low-rank structures to reduce evaluation cost.
• Enable fast, scalable pipeline selection by combining matrix/tensor factorization with greedy experiment design.
• Improve upon existing AutoML systems by achieving high performance with minimal runtime overhead

Proposed method

• Model pipeline performance as a matrix or tensor using low-rank factorization to capture dependencies across datasets, components, and hyperparameters.
• Use matrix and tensor factorization to learn embeddings of datasets and pipelines, enabling transfer learning across datasets.
• Apply greedy experiment design to select the most informative pipelines to evaluate first, minimizing regret in performance prediction.
• Leverage kernelized matrix models to extend the framework to non-linear relationships in pipeline performance.
• Integrate a polynomial runtime predictor based on dataset size and feature count to support time-constrained pipeline selection.
• Construct hyperparameter landscapes by predicting performance across hyperparameter grids without explicit optimization

Experimental results

Research questions

• RQ1 Can low-rank matrix and tensor factorization models effectively represent the combinatorial space of AutoML pipelines across diverse datasets?
• RQ2 How efficiently can greedy experiment design identify the most informative pipelines to evaluate on a new dataset?
• RQ3 To what extent can surrogate models trained on historical data predict pipeline performance on unseen datasets with minimal evaluation?
• RQ4 Can the system generalize to different estimator types and hyperparameter ranges while maintaining prediction accuracy?
• RQ5 How does the integration of runtime prediction improve time-constrained pipeline selection in practice?

Key findings

• The system achieves state-of-the-art performance in AutoML pipeline selection with under 10 seconds of experiment design time, even on large-scale datasets.
• On a 215-by-23424 error matrix, the greedy method reduced regret by 30% compared to random selection after 100 evaluations.
• The runtime predictor achieved over 70% accuracy within a factor of 2 for all estimator types, with multilayer perceptron reaching 74.5% accuracy.
• Predicted hyperparameter landscapes closely matched real landscapes, capturing both global trends and fine-grained variations with a coarse grid.
• The method required only a constant number of pipeline evaluations per new dataset, independent of the number of components.
• The system demonstrated robustness across diverse OpenML datasets, with consistent performance across different estimator types and data characteristics.