[Paper Review] BayesNAS: A Bayesian Approach for Neural Architecture Search

Hongpeng Zhou, Minghao Yang+2 more|arXiv (Cornell University)|May 13, 2019

Advanced Neural Network ApplicationsComputer Science88 citations

TL;DR

BayesNAS introduces a hierarchical automatic relevance determination prior to perform one-shot NAS, modeling dependencies between nodes to prune edges and enabling very fast search (≈0.2 GPU-days on CIFAR-10). It also supports network compression through structural sparsity.

ABSTRACT

One-Shot Neural Architecture Search (NAS) is a promising method to significantly reduce search time without any separate training. It can be treated as a Network Compression problem on the architecture parameters from an over-parameterized network. However, there are two issues associated with most one-shot NAS methods. First, dependencies between a node and its predecessors and successors are often disregarded which result in improper treatment over zero operations. Second, architecture parameters pruning based on their magnitude is questionable. In this paper, we employ the classic Bayesian learning approach to alleviate these two issues by modeling architecture parameters using hierarchical automatic relevance determination (HARD) priors. Unlike other NAS methods, we train the over-parameterized network for only one epoch then update the architecture. Impressively, this enabled us to find the architecture on CIFAR-10 within only 0.2 GPU days using a single GPU. Competitive performance can be also achieved by transferring to ImageNet. As a byproduct, our approach can be applied directly to compress convolutional neural networks by enforcing structural sparsity which achieves extremely sparse networks without accuracy deterioration.

Motivation & Objective

• Improve neural architecture search by addressing edge dependency and zero-operation bias in one-shot NAS.
• Promote sparsity and robust edge pruning via hierarchical automatic relevance determination priors.
• Achieve fast search by iterative re-weighted optimization with Hessian-based variance updates.
• Enable direct network compression through structural sparsity without major accuracy loss.

Proposed method

• Model architecture parameters with hierarchical automatic relevance determination (HARD) priors to capture dependencies among predecessors and successors.
• Represent pruning via switches and a probabilistic formulation that couples edge weights with predecessor activity.
• Train the over-parameterized network for a single epoch, then update architecture parameters using an iteratively re-weighted l1-type scheme.
• Use Laplace approximation to estimate posterior variance and guide pruning through an entropy-based criterion gamma; prune when entropy becomes nonpositive.
• Provide fast Hessian computations to enable scalable second-order updates for large networks.
• Extend to proxy tasks and network compression by adjusting the objective and pruning criteria.

Experimental results

Research questions

• RQ1 Can HARD priors properly model dependency between edges in a DAG-based NAS, ensuring connected derived graphs after pruning?
• RQ2 Does Bayesian treatment of architecture parameters outperform magnitude-based pruning in identifying zero/non-zero operations?
• RQ3 Can the approach search architectures quickly (one epoch before architecture update) while maintaining competitive accuracy on CIFAR-10 and ImageNet?
• RQ4 Can BayesNAS be used for direct network compression via structural sparsity without accuracy deterioration?

Key findings

• BayesNAS achieves CIFAR-10 architecture search within about 0.2 GPU-days on a single GPU.
• The method produces competitive CIFAR-10 architectures and transfers to ImageNet with strong performance.
• The HARD prior and entropy-based pruning preferentially zero out zero-operations and maintain connectivity.
• Extremely sparse networks with minimal or no accuracy loss are achievable as a byproduct, enabling potential compression.
• BayesNAS provides lower search costs compared to several state-of-the-art NAS methods while offering Bayesian advantages (less hyperparameter tuning, reduced overfitting).
• Pruned architectures for CIFAR-10 and ImageNet show competitive top-1/top-5 performance in mobile settings compared to manual and NAS baselines.