[Paper Review] Deeper Insights into Weight Sharing in Neural Architecture Search

Yuge Zhang, Zejun Lin+6 more|arXiv (Cornell University)|Jan 6, 2020

Advanced Neural Network ApplicationsComputer Science30 references35 citations

TL;DR

This paper empirically analyzes weight-sharing NAS, showing high instability and variance in ranking of child models, and demonstrates that partial weight sharing can stabilize rankings and improve performance.

ABSTRACT

With the success of deep neural networks, Neural Architecture Search (NAS) as a way of automatic model design has attracted wide attention. As training every child model from scratch is very time-consuming, recent works leverage weight-sharing to speed up the model evaluation procedure. These approaches greatly reduce computation by maintaining a single copy of weights on the super-net and share the weights among every child model. However, weight-sharing has no theoretical guarantee and its impact has not been well studied before. In this paper, we conduct comprehensive experiments to reveal the impact of weight-sharing: (1) The best-performing models from different runs or even from consecutive epochs within the same run have significant variance; (2) Even with high variance, we can extract valuable information from training the super-net with shared weights; (3) The interference between child models is a main factor that induces high variance; (4) Properly reducing the degree of weight sharing could effectively reduce variance and improve performance.

Motivation & Objective

• Assess how weight sharing affects NAS accuracy and stability across runs and epochs.
• Quantify variance and interference among child models in a shared-weight super-net.
• Identify mechanisms driving instability and explore strategies to reduce it.
• Propose and evaluate partial weight-sharing schemes to improve NAS performance.

Proposed method

• Construct a down-scaled NAS search space with 64 possible child models to enable ground-truth comparison.
• Train a single super-net with shared weights and evaluate all child models on a validation set.
• Compare shared-weight performance against ground-truth performance obtained by training each child model independently.
• Use Kendall’s tau to measure rank stability (S-Tau, GT-Tau) and Top-n-Rank to assess top-model discovery.
• Investigate variance sources by analyzing per-mini-batch interference among child models during super-net training.
• Experiment with partial weight sharing, including group sharing and prefix sharing, to reduce variance and study effects on ranking.

Experimental results

Research questions

• RQ1 How stable are the rankings of child models when using weight sharing across multiple runs or epochs?
• RQ2 To what extent can the super-net’s shared weights inform picking high-performing child models compared to re-training from scratch?
• RQ3 What are the main sources of variance and interference among child models under weight sharing?
• RQ4 Can reducing the degree of weight sharing (partial sharing) improve stability and alignment with ground truth performance?

Key findings

• Rank of child models under weight sharing is highly unstable across runs and epochs.
• Shared-weight training can follow but not reach ground-truth rankings, with substantial variance.
• Interference between jointly trained child models is a major contributor to ranking instability.
• Partial weight-sharing strategies (grouping, similarity-based grouping, and prefix sharing) can reduce variance and yield rankings closer to ground truth with varying computational trade-offs.
• Finetuning child models from a snapshot of the super-net can significantly improve ranking quality even with limited additional training.