[Paper Review] Neural Architecture Refinement: A Practical Way for Avoiding Overfitting in NAS

Yang Jiang, Cong Zhao+2 more|arXiv (Cornell University)|May 7, 2019

Advanced Neural Network Applications45 references4 citations

TL;DR

This paper proposes Neural Architecture Refinement (NAR), a method that alleviates overfitting in reinforcement learning-based Neural Architecture Search (NAS) by decoupling operator and skip connection optimization. Instead of jointly searching both, NAR fixes skip connections and refines only the operators, reducing search space complexity, improving reward estimation accuracy, and achieving state-of-the-art performance with fewer FLOPs and parameters across image classification and face recognition tasks.

ABSTRACT

Neural architecture search (NAS) is proposed to automate the architecture design process and attracts overwhelming interest from both academia and industry. However, it is confronted with overfitting issue due to the high-dimensional search space composed by operator selection and skip connection of each layer. This paper explores the architecture overfitting issue in depth based on the reinforcement learning-based NAS framework. We show that the policy gradient method has deep correlations with the cross entropy minimization. Based on this correlation, we further demonstrate that, though the reward of NAS is sparse, the policy gradient method implicitly assign the reward to all operations and skip connections based on the sampling frequency. However, due to the inaccurate reward estimation, curse of dimensionality problem and the hierachical structure of neural networks, reward charateristics for operators and skip connections have intrinsic differences, the assigned rewards for the skip connections are extremely noisy and inaccurate. To alleviate this problem, we propose a neural architecture refinement approach that working with an initial state-of-the-art network structure and only refining its operators. Extensive experiments have demonstrated that the proposed method can achieve fascinated results, including classification, face recognition etc.

Motivation & Objective

• To address the architecture overfitting problem in reinforcement learning-based NAS, particularly due to noisy and inaccurate reward assignment to skip connections.
• To investigate why joint optimization of operators and skip connections leads to unstable training dynamics and poor generalization.
• To improve reward estimation accuracy by decoupling the search space for operators and skip connections.
• To develop a practical, efficient NAS method that maintains or improves performance while reducing FLOPs and parameters.
• To validate the generalization capability of refined architectures across diverse computer vision tasks.

Proposed method

• Proposes Neural Architecture Refinement (NAR), which fixes skip connections and optimizes only the operator space, reducing search space dimensionality.
• Decouples the optimization of operators and skip connections, treating them as separate search domains to mitigate reward noise and instability.
• Uses a hierarchical search strategy where the controller first learns optimal operators on a fixed skip connection structure.
• Employs a proxy task with a subset of data to reduce computational cost, while ensuring accurate reward estimation through focused operator refinement.
• Leverages the correlation between policy gradient and cross-entropy minimization to stabilize training and improve convergence.
• Applies the refined architecture to downstream tasks without retraining, demonstrating generalization and efficiency.

Experimental results

Research questions

• RQ1 Why does joint optimization of operators and skip connections in NAS lead to overfitting and unstable training?
• RQ2 How does the policy gradient method implicitly assign rewards to operations and skip connections, and why is this assignment noisy for skip connections?
• RQ3 Can decoupling operator and skip connection optimization improve reward estimation accuracy and reduce overfitting in NAS?
• RQ4 Does refining only operators while keeping skip connections fixed lead to better generalization and efficiency in downstream tasks?
• RQ5 How does the hierarchical structure of neural networks affect the reward characteristics of operators versus skip connections?

Key findings

• NAR achieves 99.56% accuracy on LFW and 94.42% on CFP-FP using only 3.38×10⁷ parameters and 4.69×10⁹ FLOPs, outperforming the baseline LResNet50E-IR.
• On CIFAR-10, ResNet-18-16 achieves 93.71% accuracy with 1.41×10⁹ FLOPs, a 20% reduction in FLOPs compared to the original ResNet-18.
• The refined models generalize well to ImageNet, achieving comparable or slightly better top-1 accuracy (69.42%) than the baseline (69.39%) with fewer parameters.
• The method reduces FLOPs by up to 20% and parameters by 20% while maintaining or improving performance across multiple benchmarks.
• Experiments show that skip connections are more prone to reward noise due to high-dimensional search space, justifying the need for decoupled optimization.
• Generalization studies confirm that architectures refined on one dataset (e.g., Face-Emore V2) generalize well to others (e.g., Face-Emore V1), demonstrating robustness.