[Paper Review] Two-stage architectural fine-tuning with neural architecture search using early-stopping in image classification

Y. K. Kim, Won Joon Yun+3 more|arXiv (Cornell University)|Feb 17, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a two-stage architectural fine-tuning method that combines neural architecture search (NAS) with transfer learning for image classification, using mutation-based search space definition and early-stopping to reduce computational and search costs. The approach achieves 32.4% lower computation and 22.3% reduced search cost compared to baseline methods while improving accuracy on ResNet-18 and ResNet-50 models.

ABSTRACT

In many deep neural network (DNN) applications, the difficulty of gathering high-quality data in the industry field hinders the practical use of DNN. Thus, the concept of transfer learning has emerged, which leverages the pretrained knowledge of DNNs trained on large-scale datasets. Therefore, this paper suggests two-stage architectural fine-tuning, inspired by neural architecture search (NAS). One of main ideas is mutation, which reduces the search cost using given architectural information. Moreover, early-stopping is considered which cuts NAS costs by terminating the search process in advance. Experimental results verify our proposed method reduces 32.4% computational and 22.3% searching costs.

Motivation & Objective

• Address the challenge of data scarcity in industrial deep learning applications by enhancing transfer learning for image classification.
• Reduce the high computational and search costs associated with neural architecture search (NAS) in real-world deployment.
• Improve model performance by jointly fine-tuning both network architecture and weights using a compact, mutation-driven search space.
• Develop an early-stopping mechanism that terminates NAS before reward convergence, based on action stability in the controller.
• Demonstrate generalizability across different backbone architectures like ResNet-18 and ResNet-50 under varying search scope configurations.

Proposed method

• Define a compact search space using a base architecture (e.g., ResNet-18/50), mutation rules, and a configurable search scope to limit architectural variation.
• Apply a reinforcement learning (RL)-based controller to sample subnetworks (subnets) from the search space, with actions representing architectural choices such as filter size (3×3 or 5×5).
• Use a supernet with parameter sharing to enable efficient evaluation of candidate architectures during training.
• Implement an early-stopping strategy that halts the search when the action distribution of the controller stabilizes, even before reward convergence.
• Leverage pretrained weights from the source dataset to initialize the search, with the search scope controlling the extent of weight reuse.
• Normalize the total reward to [0,1] for consistent comparison and use test accuracy as the reward signal for the RL controller.

Experimental results

Research questions

• RQ1 Can a mutation-based search space definition reduce the search cost of NAS while maintaining or improving model performance in image classification?
• RQ2 Does early-stopping based on action distribution stability lead to significant reductions in search time and computational cost without degrading final accuracy?
• RQ3 How does the search scope influence the performance and convergence of the architectural fine-tuning process?
• RQ4 To what extent can the proposed method generalize across different backbone architectures like ResNet-18 and ResNet-50?
• RQ5 How does the use of pretrained weights in a constrained search space affect the final model accuracy compared to standard fine-tuning?

Key findings

• The proposed method reduces computational cost by 32.4% and search cost by 22.3% compared to baseline fine-tuning methods.
• The performance ranking of models across different search scopes follows the order: full > large > medium > small, indicating that broader search scopes yield better results.
• For ResNet-18, the ENASResNet-18 model with a full search scope achieved a final accuracy of 85.6%, outperforming the vanilla ResNet-18 by 4.0%.
• The early-stopping mechanism successfully reduced search time by identifying stable action distributions—e.g., at 140 subnet sampling steps for ResNet-18 with medium scope—before reward convergence.
• The action distribution stabilizes before reward convergence, validating the early-stopping strategy as both efficient and effective.
• The performance of ENASResNet-50 with full and large scopes was nearly identical, suggesting that beyond a certain point, additional architectural variation provides diminishing returns.