[Paper Review] Differentiable Architecture Search Without Training Nor Labels: A Pruning Perspective.

Miao Zhang, Steven W. Su+4 more|arXiv (Cornell University)|Jun 22, 2021

Advanced Neural Network Applications50 references4 citations

TL;DR

FreeNAS proposes a label-free and training-free neural architecture search framework by reframing NAS as a pruning-at-initialization problem. It uses a novel FreeFlow proxy to score operation importance without any training or labels, achieving state-of-the-art performance in just 3.6s and 79s on NAS-Bench-201 and DARTS search spaces, respectively.

ABSTRACT

With leveraging the weight-sharing and continuous relaxation to enable gradient-descent to alternately optimize the supernet weights and the architecture parameters through a bi-level optimization paradigm, extit{Differentiable ARchiTecture Search} (DARTS) has become the mainstream method in Neural Architecture Search (NAS) due to its simplicity and efficiency. However, more recent works found that the performance of the searched architecture barely increases with the optimization proceeding in DARTS. In addition, several concurrent works show that the NAS could find more competitive architectures without labels. The above observations reveal that the supervision signal in DARTS may be a poor indicator for architecture optimization, inspiring a foundational question: instead of using the supervision signal to perform bi-level optimization, extit{can we find high-quality architectures extbf{without any training nor labels}}? We provide an affirmative answer by customizing the NAS as a network pruning at initialization problem. By leveraging recent techniques on the network pruning at initialization, we designed a FreeFlow proxy to score the importance of candidate operations in NAS without any training nor labels, and proposed a novel framework called extit{training and label free neural architecture search} ( extbf{FreeNAS}) accordingly. We show that, without any training nor labels, FreeNAS with the proposed FreeFlow proxy can outperform most NAS baselines. More importantly, our framework is extremely efficient, which completes the architecture search within only extbf{3.6s} and extbf{79s} on a single GPU for the NAS-Bench-201 and DARTS search space, respectively. We hope our work inspires more attempts in solving NAS from the perspective of pruning at initialization.

Motivation & Objective

• To investigate whether high-quality neural architectures can be discovered without any training or labels.
• To address the poor supervision signal issue in DARTS, where performance gains plateau during optimization.
• To reframe neural architecture search as a network pruning problem at initialization.
• To develop a proxy metric that scores operation importance without requiring model training or labels.
• To achieve efficient architecture search with minimal computational cost, enabling rapid discovery of competitive architectures.

Proposed method

• Reformulate architecture search as a pruning-at-initialization problem, treating the supernet as a dense architecture to be pruned.
• Introduce FreeFlow, a novel proxy metric to score the importance of candidate operations without training or labels, based on weight magnitude and gradient flow at initialization.
• Use the FreeFlow scores to iteratively prune less important operations and select the final architecture.
• Apply a bi-level optimization paradigm not for training, but to guide pruning decisions using the FreeFlow proxy.
• Leverage existing techniques from lottery ticket and pruning-at-initialization literature to initialize and score operations efficiently.
• Optimize the architecture by selecting the top-k operations per layer based on FreeFlow scores, avoiding any backpropagation through training.

Experimental results

Research questions

• RQ1 Can high-performing neural architectures be discovered without any training or labels in neural architecture search?
• RQ2 Is the supervision signal in DARTS a reliable indicator for architecture optimization, or is it fundamentally flawed?
• RQ3 Can the principles of network pruning at initialization be effectively repurposed for architecture search?
• RQ4 Does a proxy metric like FreeFlow, computed at initialization, provide sufficient signal to guide architecture search without labels?
• RQ5 Can a training-free and label-free NAS framework achieve competitive performance with significantly reduced search time?

Key findings

• FreeNAS achieves state-of-the-art performance on both NAS-Bench-201 and DARTS search spaces without any training or labels.
• The framework completes architecture search in only 3.6 seconds on NAS-Bench-201 and 79 seconds on the DARTS search space using a single GPU.
• FreeNAS outperforms most existing NAS baselines, even those that rely on extensive training and label supervision.
• The FreeFlow proxy, computed at initialization, provides a strong and reliable signal for architecture selection without requiring backpropagation or training.
• The results demonstrate that architecture search can be effectively decoupled from training, challenging the assumption that supervision signals from training are essential for NAS.