[Paper Review] Probabilistic Neural Architecture Search

Francesco Paolo Casale, Jonathan Gordon+1 more|arXiv (Cornell University)|Feb 13, 2019

Advanced Neural Network Applications30 references70 citations

TL;DR

PARSEC introduces a memory-efficient probabilistic NAS that learns a distribution over high-performing architectures, transferable from small surrogates to larger tasks, achieving competitive results with much lower search cost.

ABSTRACT

In neural architecture search (NAS), the space of neural network architectures is automatically explored to maximize predictive accuracy for a given task. Despite the success of recent approaches, most existing methods cannot be directly applied to large scale problems because of their prohibitive computational complexity or high memory usage. In this work, we propose a Probabilistic approach to neural ARchitecture SEarCh (PARSEC) that drastically reduces memory requirements while maintaining state-of-the-art computational complexity, making it possible to directly search over more complex architectures and larger datasets. Our approach only requires as much memory as is needed to train a single architecture from our search space. This is due to a memory-efficient sampling procedure wherein we learn a probability distribution over high-performing neural network architectures. Importantly, this framework enables us to transfer the distribution of architectures learnt on smaller problems to larger ones, further reducing the computational cost. We showcase the advantages of our approach in applications to CIFAR-10 and ImageNet, where our approach outperforms methods with double its computational cost and matches the performance of methods with costs that are three orders of magnitude larger.

Motivation & Objective

• Motivate neural architecture search (NAS) that scales to large datasets without prohibitive memory usage.
• Propose a probabilistic framework to learn distributions over high-performing architectures.
• Enable transfer of architecture distributions from small surrogate problems to larger networks and datasets.
• Achieve competitive results on CIFAR-10 and ImageNet with reduced computational cost compared to peers.

Proposed method

• Define a cell-based search space (normal and reduction cells) using 7 primitive operations common in DARTS-like spaces.
• Introduce a prior p(α|π) over input/operation choices per node with independent categorical factors.
• Formulate architecture search as conditional likelihood p(y|X,v,π)=∫ p(y|X,v,α)p(α|π)dα and optimize via importance-weighted Monte Carlo empirical Bayes.
• Sample child architectures from p(α|π) and compute importance weights to update both network weights v and architecture hyper-parameters π.
• Constrain memory usage to that of training a single architecture by this sampling-based approach, enabling direct search over fully-sized networks.
• Optionally transfer the learned architecture distribution to larger networks or datasets and fine-tune on the target task.

Experimental results

Research questions

• RQ1 Can a probabilistic NAS framework learn a distribution over high-performing architectures and be trained with memory efficiency comparable to training a single architecture?
• RQ2 Does transferring architecture distributions learned on small surrogate problems to larger networks/datasets improve efficiency and performance?
• RQ3 How does PARSEC's performance and search cost compare to state-of-the-art NAS methods on CIFAR-10 and ImageNet?
• RQ4 Is fine-tuning the architecture distribution on fully-sized networks beneficial for final accuracy?

Key findings

• PARSEC matches or exceeds the performance of methods with similar search-space cost on CIFAR-10 and, after fine-tuning on large networks, improves results further.
• Memory usage is equivalent to training a single architecture, enabling direct search over fully-sized networks and larger datasets.
• Architecture distribution learned on small surrogate networks can be transferred and fine-tuned on larger problems, reducing total computational cost.
• On CIFAR-10, PARSEC with fine-tuning achieves competitive accuracy with substantially lower search cost than many baselines.
• On ImageNet, architectures identified by PARSEC outperform methods with similar search cost and approach the performance of much more expensive approaches.
• The best PARSEC network on CIFAR-10 and subsequent fine-tuning on ImageNet demonstrate strong scalability and efficiency, outperforming several contemporaries within orders of magnitude of cost.