[Paper Review] AutoML-Zero: Evolving Machine Learning Algorithms From Scratch

Esteban Real, Liang Chen+2 more|arXiv (Cornell University)|Mar 6, 2020

Machine Learning and Data Classification83 references115 citations

TL;DR

The paper presents AutoML-Zero, a framework that evolves complete ML algorithms from scratch using basic mathematical operations, discovering both simple and advanced techniques beyond neural nets, including backpropagation-like learning.

ABSTRACT

Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks---or similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the vastness of this space, evolutionary search can still discover two-layer neural networks trained by backpropagation. These simple neural networks can then be surpassed by evolving directly on tasks of interest, e.g. CIFAR-10 variants, where modern techniques emerge in the top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for the field.

Motivation & Objective

• Demonstrate that AutoML can automatically discover complete ML algorithms from basic building blocks without hand-designed components.
• Show that evolutionary search can find both simple neural nets and more advanced algorithms tailored to specific tasks.
• Investigate how evolving algorithms adapt to different task types and data regimes (e.g., few-shot, regularization).
• Provide open-source infrastructure and a scalable search strategy to explore a vast algorithm space.

Proposed method

• Represent ML algorithms as programs with three components (Setup, Predict, Learn) operating on a small memory and basic ops.
• Use an evolutionary search (regularized evolution) with mutations to build a population of candidate algorithms.
• Apply functional equivalence checking (FEC) and distributed workers to achieve high-throughput search (~10k models/sec per CPU).
• Evaluate algorithms on proxy tasks (binary CIFAR-10/MNIST-like tasks) and select via median task performance, then validate on larger tasks.
• Allow variable component lengths and a broad set of 65 ops to minimize human bias in the search space.

Experimental results

Research questions

• RQ1 Can evolutionary search discover ML algorithms from scratch using only basic mathematical operations?
• RQ2 What kinds of learning rules or architectural motifs emerge when bias is minimized in the search space?
• RQ3 How well do discovered algorithms transfer to real-world datasets beyond the proxy tasks?
• RQ4 How does the algorithm family adapt to different task types (e.g., few-shot, multi-class) and data regimes?

Key findings

• Evolutionary search can discover algorithms in a very large, generic space where random search performs poorly.
• On CIFAR-10 variants and other datasets, evolved algorithms outperform hand-designed baselines and can resemble or surpass backpropagation-trained nets.
• Algorithms exhibit techniques such as multiplicative interactions, normalized gradients, and weight averaging that emerge during evolution.
• Dropout-like regularization and learning-rate decay emerge when tasks require regularization or faster convergence, respectively.
• Starting from scratch, evolution can rediscover a neural network that uses backpropagation, and with broader ops, discovers non-neural architectures as well.