[Paper Review] A Survey on Computationally Efficient Neural Architecture Search

Shiqing Liu, Haoyu Zhang+1 more|arXiv (Cornell University)|Jun 3, 2022

Advanced Neural Network Applications4 citations

TL;DR

This survey provides a comprehensive analysis of computationally efficient neural architecture search (CE-NAS) methods, categorizing them into proxy-based and surrogate-assisted approaches to reduce training costs. It systematically compares their design principles, performance, and computational complexity, while identifying key challenges and future directions in efficient, privacy-preserving, and green AI-aware NAS.

ABSTRACT

Neural architecture search (NAS) has become increasingly popular in the deep learning community recently, mainly because it can provide an opportunity to allow interested users without rich expertise to benefit from the success of deep neural networks (DNNs). However, NAS is still laborious and time-consuming because a large number of performance estimations are required during the search process of NAS, and training DNNs is computationally intensive. To solve this major limitation of NAS, improving the computational efficiency is essential in the design of NAS. However, a systematic overview of computationally efficient NAS (CE-NAS) methods still lacks. To fill this gap, we provide a comprehensive survey of the state-of-the-art on CE-NAS by categorizing the existing work into proxy-based and surrogate-assisted NAS methods, together with a thorough discussion of their design principles and a quantitative comparison of their performances and computational complexities. The remaining challenges and open research questions are also discussed, and promising research topics in this emerging field are suggested.

Motivation & Objective

• To address the high computational cost of traditional neural architecture search (NAS) that relies on training each candidate architecture from scratch.
• To provide a systematic, up-to-date overview of computationally efficient NAS (CE-NAS) methods, filling a gap in existing literature.
• To categorize and compare proxy-based and surrogate-assisted NAS methods based on their design principles, performance, and computational complexity.
• To identify open challenges and future research directions in CE-NAS, particularly in federated learning and green AI settings.
• To guide researchers and practitioners in selecting and advancing efficient NAS methods for real-world deployment.

Proposed method

• Categorizing CE-NAS methods into proxy-based (e.g., low-fidelity estimation, one-shot NAS, network morphism) and surrogate-assisted approaches based on whether network weights are required during performance evaluation.
• Analyzing proxy-based methods that use simplified or shared weights to estimate performance without full training, reducing computational cost.
• Surveying surrogate-assisted NAS that employs performance predictors (surrogates) trained on architecture features alone, avoiding weight-dependent evaluation.
• Evaluating surrogate models using techniques like Bayesian optimization, evolutionary algorithms, and multi-objective optimization to guide architecture search efficiently.
• Integrating privacy-preserving mechanisms such as differential privacy, secure aggregation, and homomorphic encryption into federated NAS to enhance security.
• Proposing strategies for deploying NAS on resource-limited edge devices by leveraging scalable, indirect encoding and lightweight surrogate models.

Experimental results

Research questions

• RQ1 How do proxy-based and surrogate-assisted NAS methods differ in terms of computational efficiency and performance prediction accuracy?
• RQ2 What are the key design principles and trade-offs between different types of proxy-based NAS (e.g., one-shot, low-fidelity) in reducing search cost?
• RQ3 How can surrogate models be effectively trained and updated to predict the performance of unseen architectures with minimal computational overhead?
• RQ4 What are the main challenges in applying NAS under federated learning settings, particularly regarding data heterogeneity, communication cost, and hardware diversity?
• RQ5 How can NAS be made more sustainable and deployable on edge devices through green AI principles and efficient model encoding?

Key findings

• Recent CE-NAS methods have reduced search time from hundreds of GPU days to under 0.1 GPU days on CIFAR-10, demonstrating significant efficiency gains.
• Surrogate-assisted NAS methods avoid retraining by using performance predictors trained on architecture features, enabling faster evaluation without requiring network weights.
• Bayesian optimization and evolutionary algorithms are widely used in surrogate-assisted NAS, with the former offering sample efficiency and the latter enabling multi-objective optimization.
• Federated NAS remains underexplored, with challenges including non-IID data, heterogeneous hardware, and high communication costs across clients.
• Privacy-preserving techniques such as differential privacy and secure aggregation can be integrated into federated NAS to enhance security without sacrificing search efficiency.
• Deploying NAS on edge devices remains challenging due to limited computational resources, necessitating scalable, indirect encoding strategies and lightweight surrogates.