[Paper Review] FNA++: Fast Network Adaptation via Parameter Remapping and Architecture Search

Jiemin Fang, Yuzhu Sun+5 more|arXiv (Cornell University)|Jun 21, 2020

Advanced Neural Network Applications89 references4 citations

TL;DR

FNA++ proposes a fast network adaptation method that leverages parameter remapping and neural architecture search to efficiently adapt pre-trained networks (e.g., MobileNetV2) to new tasks like semantic segmentation, object detection, and pose estimation. By remapping parameters from a seed network to a super network and performing architecture search near the seed, FNA++ reduces computational cost by up to 1737× compared to SOTA NAS methods while achieving superior performance.

ABSTRACT

Deep neural networks achieve remarkable performance in many computer vision tasks. Most state-of-the-art (SOTA) semantic segmentation and object detection approaches reuse neural network architectures designed for image classification as the backbone, commonly pre-trained on ImageNet. However, performance gains can be achieved by designing network architectures specifically for detection and segmentation, as shown by recent neural architecture search (NAS) research for detection and segmentation. One major challenge though is that ImageNet pre-training of the search space representation (a.k.a. super network) or the searched networks incurs huge computational cost. In this paper, we propose a Fast Network Adaptation (FNA++) method, which can adapt both the architecture and parameters of a seed network (e.g. an ImageNet pre-trained network) to become a network with different depths, widths, or kernel sizes via a parameter remapping technique, making it possible to use NAS for segmentation and detection tasks a lot more efficiently. In our experiments, we apply FNA++ on MobileNetV2 to obtain new networks for semantic segmentation, object detection, and human pose estimation that clearly outperform existing networks designed both manually and by NAS. We also implement FNA++ on ResNets and NAS networks, which demonstrates a great generalization ability. The total computation cost of FNA++ is significantly less than SOTA segmentation and detection NAS approaches: 1737x less than DPC, 6.8x less than Auto-DeepLab, and 8.0x less than DetNAS. A series of ablation studies are performed to demonstrate the effectiveness, and detailed analysis is provided for more insights into the working mechanism. Codes are available at https://github.com/JaminFong/FNA.

Motivation & Objective

• To address the high computational cost of pre-training and searching for task-specific neural architectures in semantic segmentation and object detection.
• To enable efficient adaptation of pre-trained ImageNet models to new tasks without full retraining.
• To develop a parameter remapping technique that transfers knowledge from a seed network to a super network for faster architecture search.
• To demonstrate that architecture-level fine-tuning from a pre-trained seed outperforms random search or super network pre-training in efficiency and performance.
• To generalize the method across diverse backbones, including ResNets and NAS-generated networks.

Proposed method

• FNA++ begins with a pre-trained seed network (e.g., MobileNetV2) and expands it into a super network representing the search space.
• Parameters from the seed network are remapped to the super network using a novel parameter remapping mechanism to initialize the search process.
• Architecture search is performed on the parameter-remapped super network, focusing on architectures near the seed to reduce search space and accelerate convergence.
• The target architecture is derived based on parameter distribution in the super network, ensuring parameter initialization is informed by the seed.
• The final network is fine-tuned using the remapped parameters, enabling fast parameter adaptation with minimal training.
• The method supports both architecture-level and parameter-level adaptation, treating the process as a form of transfer learning.

Experimental results

Research questions

• RQ1 Can parameter remapping from a pre-trained seed network significantly reduce the computational cost of neural architecture search for segmentation and detection tasks?
• RQ2 Does architecture search initialized from a pre-trained seed outperform random or uniformly initialized search in terms of accuracy and efficiency?
• RQ3 How effective is the proposed parameter remapping mechanism in preserving performance during network adaptation across different tasks?
• RQ4 Can FNA++ generalize across diverse backbone architectures, including ResNets and NAS-generated models?
• RQ5 What is the impact of architectural similarity between source and target tasks on the effectiveness of architecture-level fine-tuning?

Key findings

• FNA++ reduces the total computation cost by 1737× compared to DPC, 6.8× compared to Auto-DeepLab, and 8.0× compared to DetNAS in segmentation and detection tasks.
• The adapted networks from FNA++ outperform both manually designed networks and NAS-generated models in semantic segmentation, object detection, and human pose estimation.
• The method achieves superior performance even with minimal retraining, thanks to effective parameter remapping from the seed network.
• Ablation studies confirm that parameter remapping and architecture-level fine-tuning are critical to the method’s success, with search near the seed yielding better results than random initialization.
• The generalization capability of FNA++ is validated on ResNets and NAS-generated networks, demonstrating robustness across architecture families.
• The search process is significantly accelerated because the super network is initialized with remapped parameters, eliminating the need for costly pre-training.