[Paper Review] Auto-Keras: Efficient Neural Architecture Search with Network Morphism

Haifeng Jin, Qingquan Song+1 more|arXiv (Cornell University)|Jun 27, 2018

Machine Learning and Data Classification30 references65 citations

TL;DR

This paper proposes Auto-Keras, an efficient neural architecture search framework that leverages network morphism and Bayesian optimization to reduce computational cost. By introducing a neural network kernel and tree-structured acquisition function optimization, the method enables fast, adaptive search across architectures, achieving state-of-the-art performance on benchmark datasets while supporting parallel CPU/GPU execution with dynamic memory adaptation.

ABSTRACT

Neural architecture search (NAS) has been proposed to automatically tune deep neural networks, but existing search algorithms, e.g., NASNet, PNAS, usually suffer from expensive computational cost. Network morphism, which keeps the functionality of a neural network while changing its neural architecture, could be helpful for NAS by enabling more efficient training during the search. In this paper, we propose a novel framework enabling Bayesian optimization to guide the network morphism for efficient neural architecture search. The framework develops a neural network kernel and a tree-structured acquisition function optimization algorithm to efficiently explores the search space. Intensive experiments on real-world benchmark datasets have been done to demonstrate the superior performance of the developed framework over the state-of-the-art methods. Moreover, we build an open-source AutoML system based on our method, namely Auto-Keras. The system runs in parallel on CPU and GPU, with an adaptive search strategy for different GPU memory limits.

Motivation & Objective

• To address the high computational cost of existing neural architecture search (NAS) methods like NASNet and PNAS.
• To enable efficient search in large neural architecture spaces through network morphism, preserving functionality while modifying architecture.
• To develop a Bayesian optimization framework with a neural network kernel and tree-structured acquisition function for effective exploration.
• To build a practical, open-source AutoML system (Auto-Keras) that scales across CPU and GPU with adaptive memory usage.

Proposed method

• Introduces a neural network kernel to model the relationship between morphed architectures and their performance.
• Employs a tree-structured acquisition function optimization algorithm to efficiently navigate the search space.
• Applies network morphism to transform existing architectures (e.g., by adding or removing layers) while preserving functionality.
• Uses Bayesian optimization to guide morphism steps based on performance predictions, reducing the need for full retraining.
• Designs an adaptive search strategy that adjusts based on available GPU memory, enabling efficient execution on diverse hardware.
• Implements a parallel execution pipeline for CPU and GPU to accelerate search across multiple architectures.

Experimental results

Research questions

• RQ1 Can network morphism be effectively combined with Bayesian optimization to reduce the computational cost of neural architecture search?
• RQ2 How can a neural network kernel be designed to model morphed architectures and guide search efficiently?
• RQ3 Can a tree-structured acquisition function outperform standard acquisition functions in exploring complex architecture spaces?
• RQ4 To what extent does the proposed framework achieve state-of-the-art accuracy while reducing search cost compared to existing NAS methods?
• RQ5 How well does the Auto-Keras system scale across different hardware configurations with varying GPU memory limits?

Key findings

• The proposed framework achieves state-of-the-art performance on real-world benchmark datasets, outperforming existing NAS methods in accuracy.
• The use of network morphism enables efficient architecture search with significantly reduced training cost compared to traditional NAS approaches.
• The tree-structured acquisition function optimization enables more effective exploration of the architecture space than standard methods.
• The Auto-Keras system successfully runs in parallel on CPU and GPU, adapting to different memory constraints during search.
• The framework demonstrates strong generalization and efficiency, making automated neural architecture search practical for real-world applications.