[Paper Review] Auto-Keras: An Efficient Neural Architecture Search System

Haifeng Jin, Qingquan Song+1 more|arXiv (Cornell University)|Jun 27, 2018

Machine Learning and Data Classification35 references83 citations

TL;DR

The paper presents Auto-Keras, an open-source AutoML system that uses Bayesian optimization guided by network morphism with an edit-distance neural network kernel and tree-structured acquisition optimization to efficiently search neural architectures.

ABSTRACT

Neural architecture search (NAS) has been proposed to automatically tune deep neural networks, but existing search algorithms, e.g., NASNet, PNAS, usually suffer from expensive computational cost. Network morphism, which keeps the functionality of a neural network while changing its neural architecture, could be helpful for NAS by enabling more efficient training during the search. In this paper, we propose a novel framework enabling Bayesian optimization to guide the network morphism for efficient neural architecture search. The framework develops a neural network kernel and a tree-structured acquisition function optimization algorithm to efficiently explores the search space. Intensive experiments on real-world benchmark datasets have been done to demonstrate the superior performance of the developed framework over the state-of-the-art methods. Moreover, we build an open-source AutoML system based on our method, namely Auto-Keras. The system runs in parallel on CPU and GPU, with an adaptive search strategy for different GPU memory limits.

Motivation & Objective

• Motivate and enable automated tuning of deep neural networks to reduce human expertise and computational cost.
• Develop a NAS framework that combines network morphism with Bayesian optimization to efficiently explore a large search space.
• Create an open-source AutoML system (Auto-Keras) that runs on CPU and GPU with memory-adaptive search.
• Address the non-Euclidean architecture space and tree-structured search challenges in NAS.
• Demonstrate superior performance on standard benchmarks with limited search time.

Proposed method

• Introduce an edit-distance neural network kernel to define a GP-compatible similarity between architectures.
• Define a tree-structured acquisition optimization method to navigate the NAS search space efficiently (A*/simulated annealing hybrid).
• Propose graph-level network morphism to maintain tensor shape consistency during morph operations.
• Provide a memory-adaptive strategy to fit architectures to GPU memory constraints during search.
• Implement Auto-Keras with CPU/GPU parallelism, a scalable Graph module, and a Keras-compatible API for ease of use.

Experimental results

Research questions

• RQ1 How effective is Bayesian optimization guided by network morphism in reducing NAS search time while finding high-performance architectures?
• RQ2 Can an edit-distance based neural network kernel capture architecture similarity for Gaussian processes in NAS?
• RQ3 Does the tree-structured acquisition optimization reliably balance exploration and exploitation in the morphing search space?
• RQ4 How does Auto-Keras perform on standard benchmarks (MNIST, CIFAR-10, Fashion-MNIST) compared with existing NAS/AutoML methods under constrained search time?
• RQ5 How well does the memory-adaptive strategy and CPU/GPU parallelism scale in practice?

Key findings

• Auto-Keras achieves superior performance over baseline NAS methods on MNIST, CIFAR-10, and Fashion-MNIST under limited search time.
• A novel edit-distance neural network kernel enables Gaussian process regression over non-Euclidean NAS search spaces.
• A tree-structured acquisition optimization with an A*/simulated annealing hybrid efficiently explores morphing operations.
• Graph-level network morphism preserves intermediate tensor shapes during morph, enabling scalable architecture updates.
• CPU/GPU parallelism and GPU memory adaptation substantially improve search efficiency and feasibility across varying hardware.