[论文解读] Zero-Cost Proxies for Lightweight NAS

Mohamed S. Abdelfattah, Abhinav Mehrotra+2 more|arXiv (Cornell University)|Jan 20, 2021

Advanced Neural Network Applications参考文献 38被引用 65

一句话总结

该论文提出基于在初始化阶段进行剪枝的零成本代理，用单个小批量来对神经网络进行评分，达到与减少训练的代理相当或更好的排序，并在多个基准和搜索算法上实现更快的 NAS。

ABSTRACT

Neural Architecture Search (NAS) is quickly becoming the standard methodology to design neural network models. However, NAS is typically compute-intensive because multiple models need to be evaluated before choosing the best one. To reduce the computational power and time needed, a proxy task is often used for evaluating each model instead of full training. In this paper, we evaluate conventional reduced-training proxies and quantify how well they preserve ranking between multiple models during search when compared with the rankings produced by final trained accuracy. We propose a series of zero-cost proxies, based on recent pruning literature, that use just a single minibatch of training data to compute a model's score. Our zero-cost proxies use 3 orders of magnitude less computation but can match and even outperform conventional proxies. For example, Spearman's rank correlation coefficient between final validation accuracy and our best zero-cost proxy on NAS-Bench-201 is 0.82, compared to 0.61 for EcoNAS (a recently proposed reduced-training proxy). Finally, we use these zero-cost proxies to enhance existing NAS search algorithms such as random search, reinforcement learning, evolutionary search and predictor-based search. For all search methodologies and across three different NAS datasets, we are able to significantly improve sample efficiency, and thereby decrease computation, by using our zero-cost proxies. For example on NAS-Bench-101, we achieved the same accuracy 4$ imes$ quicker than the best previous result. Our code is made public at: https://github.com/mohsaied/zero-cost-nas.

研究动机与目标

• 通过在不进行完整训练的情况下评估模型来推动减少 NAS 计算量。
• 将 pruning-at-initialization 的显著性度量改编为对整个网络进行评分以用于 NAS。
• 在多个基准上将零成本代理与传统的 reduced-training 代理进行比较。
• 将零成本代理整合到多种 NAS 算法中以提高样本效率。

提出的方法

• 将逐参数显著性指标（snip、 grasp、 synflow、 fisher、 jacob_cov）改编为通过聚合参数级分数来对整个网络进行评分。
• 使用 Spearman 相关系数对代理进行评估，相关性对比最终训练精度，在 NAS 基准上。
• 在 NAS-Bench-201 及更大基准上将零成本代理与 EcoNAS 代理进行比较。
• 探索将零成本代理整合到 NAS 算法的策略：零成本热身和零成本移动提议，覆盖 RAND、RL、AE 以及基于预测的搜索。

实验结果

研究问题

• RQ1 在不进行训练的情况下， pruning-at-initialization 的显著性度量是否可以聚合来对整个网络进行 NAS 评分？
• RQ2 零成本代理是否在 NAS 基准上比传统的 reduced-training 代理更好地保持模型排序？
• RQ3 如何将零成本代理整合到现有的 NAS 搜索算法中以提高样本效率？
• RQ4 哪种零成本度量在多样化的 NAS 空间和任务中具有鲁棒性？

主要发现

• 使用单个小批量的零成本代理可达到或超过 EcoNAS 代理的排序质量（例如，synflow 一贯表现良好）。
• 在 NAS-Bench-201 上，synflow 在各数据集上实现 Spearman ρ > 0.73，jacob_cov 也很强，多数投票集成（vote）达到 ρ > 0.8。
• 零成本代理在四种搜索算法和三个基准上实现了显著的 NAS 速度提升，在 NAS-Bench-101 上快了多达 4 倍。
• 以 synflow 为基础的零成本热身和移动提议策略显著提升 RAND、RL、AE 和预测器基础搜索的样本效率。
• synflow 在多个 NAS 基准上维持一致的顶级模型排序，其热身/移动提议在 NAS-Bench-101 和 NAS-Bench-201 上实现了最先进的结果。