[Paper Review] DC-NAS: Divide-and-Conquer Neural Architecture Search

Yunhe Wang, Yixing Xu+1 more|arXiv (Cornell University)|May 29, 2020

Advanced Neural Network Applications43 references4 citations

TL;DR

DC-NAS proposes a divide-and-conquer approach to neural architecture search that improves evaluation accuracy by clustering sub-networks based on convergence feature representations, then selecting and merging the best architecture from each cluster. It achieves 75.1% top-1 accuracy on ImageNet, outperforming state-of-the-art NAS methods under the same FLOPs constraints.

ABSTRACT

Most applications demand high-performance deep neural architectures costing limited resources. Neural architecture searching is a way of automatically exploring optimal deep neural networks in a given huge search space. However, all sub-networks are usually evaluated using the same criterion; that is, early stopping on a small proportion of the training dataset, which is an inaccurate and highly complex approach. In contrast to conventional methods, here we present a divide-and-conquer (DC) approach to effectively and efficiently search deep neural architectures. Given an arbitrary search space, we first extract feature representations of all sub-networks according to changes in parameters or output features of each layer, and then calculate the similarity between two different sampled networks based on the representations. Then, a k-means clustering is conducted to aggregate similar architectures into the same cluster, separately executing sub-network evaluation in each cluster. The best architecture in each cluster is later merged to obtain the optimal neural architecture. Experimental results conducted on several benchmarks illustrate that DC-NAS can overcome the inaccurate evaluation problem, achieving a $75.1\%$ top-1 accuracy on the ImageNet dataset, which is higher than that of state-of-the-art methods using the same search space.

Motivation & Objective

• To address the instability and inaccuracy of early stopping in conventional neural architecture search (NAS), which biases results toward faster-converging architectures.
• To reduce the evaluation complexity of large search spaces by grouping similar architectures into clusters for efficient comparison.
• To improve search performance by selecting the best architecture from each cluster and merging them to form a superior final model.
• To demonstrate that clustering based on convergence dynamics leads to better generalization and higher accuracy than uniform early stopping across all architectures.

Proposed method

• Extract feature representations of sub-networks by tracking changes in layer output features across training epochs, capturing convergence speed and behavior.
• Compute similarity between architectures using the extracted feature representations to group similar architectures together.
• Apply k-means clustering to partition the search space into K clusters based on architectural similarity in convergence dynamics.
• Select the best-performing sub-network from each cluster using a performance metric based on validation accuracy.
• Merge the top-performing models from each cluster to form the final searched architecture.
• Use a super-network with learned operation probabilities to sample 10 million architectures for feature extraction and clustering.

Experimental results

Research questions

• RQ1 Can clustering architectures based on convergence dynamics improve the reliability of NAS evaluation compared to uniform early stopping?
• RQ2 Does dividing the search space into clusters of similar architectures lead to better final model performance?
• RQ3 Can the divide-and-conquer strategy reduce the risk of selecting suboptimal architectures due to early stopping bias?
• RQ4 How does the performance of DC-NAS compare to state-of-the-art NAS methods in terms of accuracy, FLOPs, and latency?

Key findings

• DC-NAS achieves 75.1% top-1 accuracy on the ImageNet validation set, setting a new state-of-the-art among hardware-efficient NAS models.
• The method improves search performance by 1.5–2.0% over baseline methods like FBNet under similar FLOPs constraints.
• The total search cost is 231 GPU hours, only 1.07× higher than the baseline FBNet, indicating minimal overhead.
• The latency of the best DC-NAS model (118.12ms on ARM) is comparable to FBNet-C (116.55ms), showing strong hardware efficiency.
• Increasing the number of clusters (K) beyond 5 yields no significant performance gain, indicating K=5 is sufficient for effective clustering.
• The method demonstrates that convergence behavior, not just architecture structure, is a strong indicator of final performance, reducing early stopping bias.