[Paper Review] AutoHAS: Differentiable Hyper-parameter and Architecture Search.

Xuanyi Dong, Mingxing Tan+4 more|arXiv (Cornell University)|Jun 5, 2020

Advanced Neural Network Applications44 references17 citations

TL;DR

AutoHAS proposes a differentiable approach to jointly search for optimal neural network architectures and hyper-parameters by modeling continuous hyper-parameters as a linear combination of categorical basis vectors, enabling efficient weight sharing across all configurations. It achieves up to 2% accuracy gain on ImageNet and 0.4 F1 improvement on SQuAD 1.1 with search costs comparable to training a single model, outperforming random search and Bayesian optimization by 10x in compute efficiency.

ABSTRACT

Neural Architecture Search (NAS) has achieved significant progress in pushing state-of-the-art performance. While previous NAS methods search for different network architectures with the same hyper-parameters, we argue that such search would lead to sub-optimal results. We empirically observe that different architectures tend to favor their own hyper-parameters. In this work, we extend NAS to a broader and more practical space by combining hyper-parameter and architecture search. As architecture choices are often categorical whereas hyper-parameter choices are often continuous, a critical challenge here is how to handle these two types of values in a joint search space. To tackle this challenge, we propose AutoHAS, a differentiable hyper-parameter and architecture search approach, with the idea of discretizing the continuous space into a linear combination of multiple categorical basis. A key element of AutoHAS is the use of weight sharing across all architectures and hyper-parameters which enables efficient search over the large joint search space. Experimental results on MobileNet/ResNet/EfficientNet/BERT show that AutoHAS significantly improves accuracy up to 2% on ImageNet and F1 score up to 0.4 on SQuAD 1.1, with search cost comparable to training a single model. Compared to other AutoML methods, such as random search or Bayesian methods, AutoHAS can achieve better accuracy with 10x less compute cost.

Motivation & Objective

• To address the limitation of prior NAS methods that fix hyper-parameters across architectures, which leads to sub-optimal performance.
• To enable joint search over both discrete architectures and continuous hyper-parameters in a unified, differentiable framework.
• To reduce search cost to near-single-model training levels while maintaining high accuracy improvements.
• To improve model performance on vision and NLP benchmarks by jointly optimizing architecture and hyper-parameters.
• To develop a method that efficiently explores the large joint search space of architectures and hyper-parameters using weight sharing and differentiable operations.

Proposed method

• AutoHAS discretizes the continuous hyper-parameter space into a linear combination of categorical basis vectors, enabling differentiable optimization over both continuous and discrete search spaces.
• It formulates the joint search space as a combination of architectural choices (categorical) and hyper-parameters (continuous), using a differentiable relaxation to allow gradient-based optimization.
• The method employs weight sharing across all architecture and hyper-parameter configurations, drastically reducing the computational cost of evaluating the joint space.
• A differentiable architecture search is performed using a continuous relaxation of the architecture space, while hyper-parameters are optimized via gradient descent on the basis vector coefficients.
• The approach uses a single differentiable search process that jointly optimizes architecture and hyper-parameters through end-to-end backpropagation.
• The final model is selected from the best-performing architecture-hyper-parameter configuration found during the differentiable search.

Experimental results

Research questions

• RQ1 Can joint optimization of neural architecture and hyper-parameters lead to better model performance than separate or fixed-hyper-parameter NAS?
• RQ2 How can continuous hyper-parameters be effectively integrated into a differentiable search framework that also handles categorical architecture choices?
• RQ3 What is the computational cost of joint architecture and hyper-parameter search, and can it be reduced to near-single-model training levels?
• RQ4 How does AutoHAS compare in accuracy and efficiency to random search and Bayesian optimization in hyper-parameter and architecture search?
• RQ5 To what extent do different architectures benefit from distinct hyper-parameter configurations, and can this be captured in a unified search space?

Key findings

• AutoHAS improves top-1 accuracy by up to 2% on ImageNet compared to standard NAS methods that fix hyper-parameters.
• On the SQuAD 1.1 benchmark, AutoHAS achieves a 0.4 absolute improvement in F1 score over baseline models.
• The method achieves these gains with a search cost comparable to training a single model, making it highly efficient.
• AutoHAS reduces the required compute by 10x compared to random search and Bayesian optimization while achieving better accuracy.
• The joint search space enables architectures to discover hyper-parameters that are specifically tailored to their structure, leading to improved generalization.
• Weight sharing across all configurations enables scalable and efficient exploration of the large joint search space without significant overhead.