[Paper Review] ROME: Robustifying Memory-Efficient NAS via Topology Disentanglement and Gradients Accumulation

Xiaoxing Wang, Xiangxiang Chu+5 more|arXiv (Cornell University)|Nov 23, 2020

Domain Adaptation and Few-Shot Learning40 references4 citations

TL;DR

ROME proposes a robust, memory-efficient neural architecture search method that addresses instability in single-path differentiable NAS by decoupling topology from operations via parameter disentanglement and using gradient accumulation to reduce sampling variance. The method achieves state-of-the-art performance across multiple benchmarks with improved search stability and consistency.

ABSTRACT

Single-path based differentiable neural architecture search has great strengths for its low computational cost and memory-friendly nature. However, we surprisingly discover that it suffers from severe searching instability which has been primarily ignored, posing a potential weakness for a wider application. In this paper, we delve into its performance collapse issue and propose a new algorithm called RObustifying Memory-Efficient NAS (ROME). Specifically, 1) for consistent topology in the search and evaluation stage, we involve separate parameters to disentangle the topology from the operations of the architecture. In such a way, we can independently sample connections and operations without interference; 2) to discount sampling unfairness and variance, we enforce fair sampling for weight update and apply a gradient accumulation mechanism for architecture parameters. Extensive experiments demonstrate that our proposed method has strong performance and robustness, where it mostly achieves state-of-the-art results on a large number of standard benchmarks.

Motivation & Objective

• To address the overlooked issue of searching instability in single-path differentiable NAS, which undermines its reliability despite low memory usage.
• To ensure consistent architecture topology between search and evaluation phases by decoupling connection and operation parameters.
• To reduce sampling variance and unfairness in weight updates during architecture optimization.
• To improve the robustness and performance consistency of memory-efficient NAS through gradient accumulation and topology disentanglement.

Proposed method

• Introduces separate parameters to disentangle the architecture's topology (connections) from its operations, enabling independent sampling of connections and operations.
• Employs a gradient accumulation mechanism to stabilize updates of architecture parameters by averaging gradients over multiple sampling steps.
• Enforces fair sampling of architecture parameters by distributing updates uniformly across different connection configurations.
• Maintains a single-path search path to preserve memory efficiency while improving stability through architectural decoupling.
• Uses differentiable search with gradient-based optimization, but modifies the update process to reduce variance and bias in architecture parameter updates.

Experimental results

Research questions

• RQ1 How does topology entanglement in single-path NAS contribute to search instability?
• RQ2 Can parameter disentanglement between connections and operations improve search consistency?
• RQ3 To what extent does gradient accumulation reduce sampling variance in architecture parameter updates?
• RQ4 Does the proposed method achieve better performance and robustness compared to existing memory-efficient NAS approaches?

Key findings

• ROME achieves state-of-the-art performance on a wide range of standard benchmarks, demonstrating superior search stability and consistency.
• The method significantly reduces search instability caused by entangled topology and operations in single-path NAS.
• Gradient accumulation effectively mitigates sampling variance and improves the fairness of architecture parameter updates.
• Topology disentanglement enables independent sampling of connections and operations without interference, enhancing training stability.