[Paper Review] Neural Architecture Search with Bayesian Optimisation and Optimal Transport

Kirthevasan Kandasamy, Willie Neiswanger+3 more|arXiv (Cornell University)|Feb 11, 2018

Machine Learning and Algorithms50 references265 citations

TL;DR

NASBOT uses Bayesian optimisation over neural architectures with a novel OTMANN distance via optimal transport and an evolutionary strategy to optimize the acquisition, outperforming baselines on MLP/CNN model selection tasks.

ABSTRACT

Bayesian Optimisation (BO) refers to a class of methods for global optimisation of a function $f$ which is only accessible via point evaluations. It is typically used in settings where $f$ is expensive to evaluate. A common use case for BO in machine learning is model selection, where it is not possible to analytically model the generalisation performance of a statistical model, and we resort to noisy and expensive training and validation procedures to choose the best model. Conventional BO methods have focused on Euclidean and categorical domains, which, in the context of model selection, only permits tuning scalar hyper-parameters of machine learning algorithms. However, with the surge of interest in deep learning, there is an increasing demand to tune neural network \emph{architectures}. In this work, we develop NASBOT, a Gaussian process based BO framework for neural architecture search. To accomplish this, we develop a distance metric in the space of neural network architectures which can be computed efficiently via an optimal transport program. This distance might be of independent interest to the deep learning community as it may find applications outside of BO. We demonstrate that NASBOT outperforms other alternatives for architecture search in several cross validation based model selection tasks on multi-layer perceptrons and convolutional neural networks.

Motivation & Objective

• Motivate efficient neural architecture search due to expensive network evaluations.
• Introduce a distance metric for architectures (OTMANN) computed via optimal transport.
• Develop NASBOT, a Gaussian-process based BO framework for architectures.
• Demonstrate superior performance of NASBOT over baselines on multiple datasets.

Proposed method

• Define neural architectures as graphs with layer masses and path-length based structural information.
• Propose OTMANN distance: a minimisation over mass transport between layers incorporating label mismatch, non-assignment, and structural penalties.
• Formulate OTMANN as an Optimal Transport program with a kernel ^{- eta d^p} to plug into Gaussian processes.
• Construct NASBOT: a BO method using a GP prior and EI acquisition with an evolutionary algorithm to optimise the acquisition.
• Use an augmentation of the architecture space via mutations to explore candidate architectures.
• Provide practical implementations and discuss extensions to scale to large search spaces.

Experimental results

Research questions

• RQ1 How can Bayesian optimisation be effectively performed over discrete neural architectures?
• RQ2 Does NASBOT with OTMANN outperform baselines (RAND, EA, TreeBO) in architecture search across tasks?
• RQ3 What characteristics of architectural distance (OTMANN) correlate with generalisation performance?
• RQ4 Can NASBOT handle both MLP and CNN search spaces and scale with parallel evaluations?

Key findings

• NASBOT consistently outperforms RAND, EA, and TreeBO baselines across multiple regression and classification datasets in cross-validation results.
• NASBOT achieves the best test performance on several datasets (e.g., Blogs, Indoor, Slice, Naval, Protein, News, CIFAR-10) in the reported experiments.
• The approach demonstrates efficient navigation of the architecture space and faster convergence to high-performing models compared to baselines.
• The OTMANN distance provides meaningful similarity assessments between architectures, enabling effective GP-based BO for architecture search.
• The authors provide Python implementations of OTMANN and NASBOT.
• On CIFAR-10, NASBOT found competitive models that were further trained to demonstrate strong test performance.