[Paper Review] PC-DARTS: Partial Channel Connections for Memory-Efficient Differentiable Architecture Search.

Yuhui Xu, Lingxi Xie+5 more|arXiv (Cornell University)|Jul 12, 2019

Machine Learning and ELMComputer Science125 citations

TL;DR

PC-DARTS introduces a memory-efficient differentiable architecture search method by sampling only a subset of channels in the super-network during search, reducing computational redundancy. By applying edge normalization to stabilize edge selection, it enables faster, more stable training with larger batch sizes, achieving 2.57% test error on CIFAR-10 in 0.1 GPU-days and 24.2% top-1 error on ImageNet (mobile setting) in 3.8 GPU-days.

ABSTRACT

Differentiable architecture search (DARTS) provided a fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training a super-net and search for an optimal architecture. In this paper, we present a novel approach, namely Partially-Connected DARTS, by sampling a small part of super-net to reduce the redundancy in network space, thereby performing a more efficient search without comprising the performance. In particular, we perform operation search in a subset of channels and leave the held out part unchanged. This strategy may suffer from an undesired inconsistency on selecting the edges of super-net caused by the sampling of different channels. We solve it by introducing edge normalization, which adds a new set of edge-level hyper-parameters during search to reduce uncertainty in search. Thanks to the reduced memory cost, PC-DARTS can be trained with a larger batch size and, consequently, enjoys both faster speed and higher training stability. Experimental results demonstrate the effectiveness of the proposed method. Specifically, we achieve an error rate of 2:57% on CIFAR10 within merely 0:1 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24:2% on ImageNet (under the mobile setting) within 3.8 GPU-days for search. We have made our code available: https://github.com/yuhuixu1993/PC-DARTS.

Motivation & Objective

• To reduce the high memory and computational cost of standard DARTS during architecture search.
• To address the instability in edge selection caused by sampling only a subset of channels in the super-network.
• To maintain high search performance while significantly lowering memory consumption.
• To enable larger batch sizes during training for improved training stability and speed.

Proposed method

• PC-DARTS samples only a subset of channels in the super-network during the search phase, reducing redundancy and memory usage.
• Operation search is performed on a selected subset of channels, while the remaining channels are kept unchanged to preserve feature representation.
• Edge normalization is introduced as a set of learnable edge-level hyper-parameters to reduce uncertainty and inconsistency in architecture search decisions.
• The method maintains the differentiable search paradigm but applies it selectively to a fraction of channels, improving efficiency without sacrificing performance.
• The training process leverages larger batch sizes due to reduced memory footprint, enhancing training stability and convergence speed.

Experimental results

Research questions

• RQ1 Can sampling only a subset of channels in the super-network reduce memory consumption in differentiable architecture search without degrading performance?
• RQ2 How can edge selection inconsistency be mitigated when only a portion of channels is used during search?
• RQ3 Does reducing the search space via partial connections lead to faster convergence and improved training stability?
• RQ4 To what extent can PC-DARTS achieve state-of-the-art performance on benchmark datasets like CIFAR-10 and ImageNet under memory and time constraints?

Key findings

• PC-DARTS achieves a top-1 error rate of 2.57% on CIFAR-10 with only 0.1 GPU-days of search time.
• On ImageNet under the mobile setting, PC-DARTS reaches a state-of-the-art top-1 error rate of 24.2% within 3.8 GPU-days of search.
• The method enables larger batch sizes due to reduced memory usage, resulting in faster training and improved stability.
• Edge normalization effectively reduces uncertainty in edge selection, improving search consistency despite partial channel sampling.
• The proposed approach maintains high performance while significantly reducing the computational and memory overhead of standard DARTS.