[논문 리뷰] PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search

Yuhui Xu, Lingxi Xie외 5명|arXiv (Cornell University)|2019. 07. 12.

Advanced Neural Network Applications참고 문헌 43인용 수 389

한 줄 요약

PC-DARTS는 채널 샘플링과 엣지 정규화를 도입하여 differentiable NAS의 메모리와 계산량을 줄이고, CIFAR-10에서 0.1 GPU-days로 2.57% 오차를, ImageNet(mobile)에서 3.8 GPU-days로 24.2% top-1 오차를 달성한다.

ABSTRACT

Differentiable architecture search (DARTS) provided a fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training a super-network and searching for an optimal architecture. In this paper, we present a novel approach, namely, Partially-Connected DARTS, by sampling a small part of super-network to reduce the redundancy in exploring the network space, thereby performing a more efficient search without comprising the performance. In particular, we perform operation search in a subset of channels while bypassing the held out part in a shortcut. This strategy may suffer from an undesired inconsistency on selecting the edges of super-net caused by sampling different channels. We alleviate it using edge normalization, which adds a new set of edge-level parameters to reduce uncertainty in search. Thanks to the reduced memory cost, PC-DARTS can be trained with a larger batch size and, consequently, enjoys both faster speed and higher training stability. Experimental results demonstrate the effectiveness of the proposed method. Specifically, we achieve an error rate of 2.57% on CIFAR10 with merely 0.1 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.2% on ImageNet (under the mobile setting) using 3.8 GPU-days for search. Our code has been made available at: https://github.com/yuhuixu1993/PC-DARTS.

연구 동기 및 목표

• Motivate reducing memory and computation overhead in differentiable architecture search (DARTS).
• Introduce a partial channel connection scheme to lower memory usage during search.
• Propose edge normalization to stabilize architecture selection under channel sampling.
• Demonstrate effectiveness on CIFAR-10 and ImageNet, including direct ImageNet search.

제안 방법

• Perform operation search on a subset of channels (1/K of channels) while bypassing the rest via a shortcut, reducing memory by approximately a factor K.
• Introduce edge-level parameters beta_{i,j} to normalize edge contributions and stabilize selection across sampled channels.
• Combine channel-sampled operation outputs with the usual DARTS formulation by stacking the normalized edge and operation weights.
• Provide GPU-friendly implementation by shuffling channels to maintain efficiency.
• Allow larger batch sizes during search, improving speed and stability.
• Evaluate using the standard DARTS search space with eight candidate operations (e.g., separable convolutions, dilated convolutions, pooling, skip, zero).

실험 결과

연구 질문

• RQ1 Can partial channel connections reduce memory and enable larger batch sizes without sacrificing accuracy?
• RQ2 Does edge normalization stabilize edge selection when channel sampling is used in architecture search?
• RQ3 How do PC-DARTS variants perform on CIFAR-10 and ImageNet compared to DARTS and other NAS methods?
• RQ4 Is direct ImageNet search feasible with memory-efficient NAS, and what are the resulting architectures and accuracies?

주요 결과

• PC-DARTS achieves 2.57% error on CIFAR-10 with 0.1 GPU-days of search time.
• PC-DARTS achieves 24.2% top-1 error on ImageNet (mobile setting) with 3.8 GPU-days of search.
• Channel sampling reduces memory by about K× and enables 4× larger batch sizes in CIFAR-10 experiments.
• Edge normalization stabilizes architecture search and improves robustness across runs and hyper-parameter variations.
• PC-DARTS outperforms the DARTS baseline on CIFAR-10 (2.57% vs 2.76% error) and enables direct ImageNet search where DARTS was unstable.
• The method supports efficient, memory-conscious NAS with competitive or superior accuracy and faster search times.