[Paper Review] Multi-Objective Neural Architecture Search Based on Diverse Structures and Adaptive Recommendation

Chunnan Wang, Hongzhi Wang+2 more|arXiv (Cornell University)|Jul 6, 2020

Advanced Multi-Objective Optimization Algorithms35 references4 citations

TL;DR

This paper proposes MoARR, a multi-objective neural architecture search method that leverages high-performing, pre-discovered cell structures to enable diverse and flexible network architectures while reducing search space. By adaptively learning a reverse recommendation model from historical evaluation data, MoARR efficiently identifies Pareto-optimal architectures with high accuracy and low parameter count, achieving 1.9% error rate and 2.3M parameters on CIFAR-10 in just 6 GPU hours—surpassing state-of-the-art methods.

ABSTRACT

The search space of neural architecture search (NAS) for convolutional neural network (CNN) is huge. To reduce searching cost, most NAS algorithms use fixed outer network level structure, and search the repeatable cell structure only. Such kind of fixed architecture performs well when enough cells and channels are used. However, when the architecture becomes more lightweight, the performance decreases significantly. To obtain better lightweight architectures, more flexible and diversified neural architectures are in demand, and more efficient methods should be designed for larger search space. Motivated by this, we propose MoARR algorithm, which utilizes the existing research results and historical information to quickly find architectures that are both lightweight and accurate. We use the discovered high-performance cells to construct network architectures. This method increases the network architecture diversity while also reduces the search space of cell structure design. In addition, we designs a novel multi-objective method to effectively analyze the historical evaluation information, so as to efficiently search for the Pareto optimal architectures with high accuracy and small parameter number. Experimental results show that our MoARR can achieve a powerful and lightweight model (with 1.9% error rate and 2.3M parameters) on CIFAR-10 in 6 GPU hours, which is better than the state-of-the-arts. The explored architecture is transferable to ImageNet and achieves 76.0% top-1 accuracy with 4.9M parameters.

Motivation & Objective

• To address the limitations of fixed outer network structures in NAS, which degrade performance when models are made lightweight.
• To reduce the search cost in large, complex search spaces while increasing architectural diversity and flexibility.
• To develop an efficient multi-objective optimization method that balances accuracy and parameter count for lightweight CNNs.
• To leverage existing high-performing cell structures to accelerate search and improve model quality.
• To design a dynamic recommendation mechanism that learns from historical evaluations to guide future architecture selection.

Proposed method

• The method constructs flexible network architectures by reusing high-performing, pre-discovered cell structures from prior NAS work.
• It introduces a novel multi-objective optimization algorithm, MoARR, which uses historical evaluation data to learn a reverse recommendation model (RRModel).
• The RRModel adaptively recommends architectures with higher accuracy and lower parameter counts by modeling relationships between architecture encoding, accuracy, and parameter count.
• MoARR iteratively evaluates recommended architectures, refining the RRModel with each new feedback to improve recommendation quality over time.
• The approach reduces unnecessary evaluations by avoiding poor-performing architectures, thus accelerating convergence to Pareto-optimal solutions.
• The method is designed to be efficient and scalable, particularly suited for expensive, multi-objective NAS problems with high evaluation cost.

Experimental results

Research questions

• RQ1 Can reusing high-performing, pre-discovered cell structures significantly reduce the search space and improve efficiency in neural architecture search?
• RQ2 How can historical evaluation data be effectively leveraged to guide the search toward Pareto-optimal architectures with high accuracy and low parameter count?
• RQ3 Can an adaptive reverse recommendation model outperform traditional multi-objective optimization methods in NAS with expensive evaluation costs?
• RQ4 To what extent does architectural diversity from varied cell stacking improve model performance in lightweight settings?
• RQ5 Can the proposed method achieve state-of-the-art performance on benchmark datasets like CIFAR-10 and ImageNet with minimal computational resources?

Key findings

• MoARR achieved a 1.9% error rate with only 2.3 million parameters on CIFAR-10, outperforming existing state-of-the-art methods.
• The search process required only 6 GPU hours, significantly reducing training cost compared to prior approaches.
• The discovered architecture transferred well to ImageNet, achieving 76.0% top-1 accuracy with 4.9 million parameters.
• The use of a reverse recommendation model based on historical data reduced the number of ineffective evaluations, improving search efficiency.
• The method demonstrated superior performance by combining architectural diversity with adaptive, data-driven recommendation.
• The approach is transferable beyond CNNs and can be extended to other neural network types such as GNNs and RNNs.