[论文解读] Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search

Yibo Yang, Shan You+4 more|arXiv (Cornell University)|Jan 27, 2021

Advanced Neural Network Applications参考文献 55被引用 8

一句话总结

该论文提出EnTranNAS，一种可微神经架构搜索框架，通过将搜索与评估解耦，利用Engine-cell和Transit-cell，提升了一致性、效率与灵活性。该方法支持可微稀疏化与拓扑学习，在CIFAR-10上仅需0.06 GPU-day，在ImageNet上仅需2.1 GPU-day，即可实现最先进性能，同时支持灵活、无约束的架构拓扑。

ABSTRACT

Most differentiable neural architecture search methods construct a super-net for search and derive a target-net as its sub-graph for evaluation. There exists a significant gap between the architectures in search and evaluation. As a result, current methods suffer from an inconsistent, inefficient, and inflexible search process. In this paper, we introduce EnTranNAS that is composed of Engine-cells and Transit-cells. The Engine-cell is differentiable for architecture search, while the Transit-cell only transits a sub-graph by architecture derivation. Consequently, the gap between the architectures in search and evaluation is significantly reduced. Our method also spares much memory and computation cost, which speeds up the search process. A feature sharing strategy is introduced for more balanced optimization and more efficient search. Furthermore, we develop an architecture derivation method to replace the traditional one that is based on a hand-crafted rule. Our method enables differentiable sparsification, and keeps the derived architecture equivalent to that of Engine-cell, which further improves the consistency between search and evaluation. Besides, it supports the search for topology where a node can be connected to prior nodes with any number of connections, so that the searched architectures could be more flexible. For experiments on CIFAR-10, our search on the standard space requires only 0.06 GPU-day. We further have an error rate of 2.22% with 0.07 GPU-day for the search on an extended space. We can also directly perform the search on ImageNet with topology learnable and achieve a top-1 error rate of 23.8% in 2.1 GPU-day.

研究动机与目标

• 为解决由于架构间隙导致的可微NAS中搜索与评估之间的一致性问题。
• 通过解耦搜索与推导，降低超网络训练中的内存与计算开销。
• 实现无需手工设计拓扑规则的可微、灵活架构搜索。
• 通过Transit-cell之间的特征共享策略，改善优化平衡。
• 通过可微稀疏化，支持每个节点具有任意连接性的拓扑学习。

提出的方法

• 引入Engine-cell用于可微架构搜索，引入Transit-cell用于轻量化、无参数的架构推导。
• 采用特征共享策略，在搜索过程中平衡参数更新并减少内存使用。
• 提出一种可微稀疏化方法，确保推导出的架构与搜索阶段的架构保持等价。
• 使用可学习的架构参数化方法，实现端到端的拓扑训练，替代如top-k选择等固定规则。
• 引入EnTranNAS-DST，采用温度退火的Gumbel-Softmax与稀疏性正则化损失，学习灵活的拓扑结构。
• 采用渐进式稀疏化策略，在搜索过程中自适应地丢弃连接，同时在保持性能的前提下降低计算量。

实验结果

研究问题

• RQ1 能否通过减少搜索阶段与评估阶段架构之间的差距，来提升可微NAS中的一致性？
• RQ2 能否在不牺牲搜索质量的前提下，显著降低超网络训练的计算与内存开销？
• RQ3 可微架构搜索能否支持灵活的拓扑结构，使得节点可与任意数量的前序节点连接？
• RQ4 可微稀疏化是否能比手工设计的剪枝规则（如top-k选择）带来更好的泛化性能？
• RQ5 端到端可微拓扑学习能否在搜索效率与准确率上超越现有方法？

主要发现

• 在扩展的搜索空间上搜索时，EnTranNAS在CIFAR-10上仅需0.07 GPU-day，即可实现2.22%的top-1错误率。
• 在CIFAR-10上，EnTranNAS仅需0.06 GPU-day即可完成标准空间搜索，相比之前方法显著降低了搜索成本。
• EnTranNAS直接在ImageNet上进行搜索，仅用2.1 GPU-day即实现24.3%的top-1错误率，优于搜索成本更高的方法。
• EnTranNAS-DST在ImageNet上实现23.8%的top-1错误率，仅用2.1 GPU-day，准确率超过EnTranNAS与PC-DARTS，同时支持无约束的拓扑学习。
• 消融实验表明，控制超参数λ可实现具有多样化容量与拓扑结构的架构搜索。
• 该方法展现出卓越的效率与一致性，由于可微稀疏化，推导出的架构与搜索阶段的架构保持等价。