[Paper Review] AutoML using Metadata Language Embeddings

Iddo Drori, Lu Liu+6 more|arXiv (Cornell University)|Oct 8, 2019

Topic Modeling15 references17 citations

TL;DR

This paper proposes a meta-AutoML framework that enhances automated machine learning by leveraging NLP embeddings of dataset metadata and machine learning pipeline documentation. By training a neural network to predict the similarity between optimal pipelines based on embedded dataset and algorithm descriptions, the system achieves zero-shot AutoML performance competitive with state-of-the-art tools—delivering near-optimal results in under one second, outperforming OBOE, AutoSklearn, AlphaD3M, and TPOT under a one-minute computation limit.

ABSTRACT

As a human choosing a supervised learning algorithm, it is natural to begin by reading a text description of the dataset and documentation for the algorithms you might use. We demonstrate that the same idea improves the performance of automated machine learning methods. We use language embeddings from modern NLP to improve state-of-the-art AutoML systems by augmenting their recommendations with vector embeddings of datasets and of algorithms. We use these embeddings in a neural architecture to learn the distance between best-performing pipelines. The resulting (meta-)AutoML framework improves on the performance of existing AutoML frameworks. Our zero-shot AutoML system using dataset metadata embeddings provides good solutions instantaneously, running in under one second of computation. Performance is competitive with AutoML systems OBOE, AutoSklearn, AlphaD3M, and TPOT when each framework is allocated a minute of computation. We make our data, models, and code publicly available.

Motivation & Objective

• To improve AutoML performance by incorporating human-readable dataset descriptions and algorithm documentation into automated pipeline selection.
• To develop a zero-shot AutoML system that recommends high-performing pipelines instantly using semantic embeddings of datasets and algorithms.
• To create a new benchmark dataset, AutoKaggle, containing structured metadata, tasks, and executable pipelines from Kaggle competitions.
• To learn a metric space where similar datasets and optimal pipelines are embedded close together, enabling transfer learning across datasets.
• To demonstrate that NLP-based embeddings of unstructured text (e.g., dataset titles, descriptions, and algorithm docs) can significantly improve AutoML recommendation accuracy.

Proposed method

• The method uses pre-trained language models to generate dense vector embeddings from free-text dataset metadata (title, description, keywords) and algorithm documentation.
• It constructs a neural network that learns a metric on dataset embeddings by minimizing the distance between embeddings of datasets whose best-performing pipelines are similar.
• The system uses a zero-shot inference strategy: for a new dataset, it computes its metadata embedding and retrieves the best pipeline known for the nearest training dataset.
• It introduces a pipeline embedding space where each pipeline is represented as a vector derived from its constituent components and documentation.
• A metric learning objective is defined to align the distance between dataset embeddings with the distance between the embeddings of their respective best-performing pipelines.
• The framework is evaluated using the AutoKaggle dataset, a curated collection of Kaggle competition metadata, tasks, and executable pipelines translated into Scikit-learn-compatible formats.

Experimental results

Research questions

• RQ1 Can NLP embeddings of dataset metadata improve zero-shot AutoML performance compared to traditional meta-features?
• RQ2 Can embeddings of machine learning pipeline documentation enhance the generalization of AutoML systems across diverse datasets?
• RQ3 Does learning a joint metric space between dataset embeddings and pipeline embeddings lead to better transfer performance in AutoML?
• RQ4 How does the performance of a zero-shot AutoML system using metadata embeddings compare to state-of-the-art AutoML frameworks under time constraints?
• RQ5 To what extent do human-generated winning pipelines on Kaggle align with the recommendations derived from learned embeddings?

Key findings

• The zero-shot AutoML system using only dataset metadata embeddings (Ours DE) achieves an average accuracy of 0.87 across 9 tabular classification datasets, running in under one second.
• The pipeline embedding-based system (Ours PE) improves performance over baseline AutoML tools, achieving 0.90 accuracy on the HR dataset and 0.87 on Titanic, matching or exceeding human-generated pipelines.
• On the Seattle dataset, the system achieves 1.00 accuracy in zero-shot mode, matching the performance of OBOE and TPOT under a one-minute time budget.
• The method outperforms OBOE, AutoSklearn, AlphaD3M, and TPOT on 6 out of 9 datasets when all are given one minute of computation time.
• The neural metric learning framework successfully captures semantic similarity between datasets and optimal pipelines, enabling accurate zero-shot recommendations based on textual metadata alone.
• The AutoKaggle dataset, containing 100+ Kaggle competition solutions in executable Scikit-learn format, enables reproducible evaluation and benchmarking of AutoML systems.