[Paper Review] Multi-objective Neural Architecture Search via Non-stationary Policy Gradient

Zewei Chen, Fengwei Zhou+2 more|arXiv (Cornell University)|Jan 23, 2020

Advanced Multi-Objective Optimization Algorithms48 references6 citations

TL;DR

This paper proposes NPG-NAS, a reinforcement learning framework using non-stationary policy gradients to efficiently approximate the full Pareto front in multi-objective neural architecture search. By introducing adaptive reward functions (ADF and ADC) and a cosine temperature decay for exploration, it achieves fast convergence and high accuracy, outperforming existing methods on CIFAR-10, CIFAR-100, and ImageNet with superior Pareto coverage and state-of-the-art performance at reduced model size.

ABSTRACT

Multi-objective Neural Architecture Search (NAS) aims to discover novel architectures in the presence of multiple conflicting objectives. Despite recent progress, the problem of approximating the full Pareto front accurately and efficiently remains challenging. In this work, we explore the novel reinforcement learning (RL) based paradigm of non-stationary policy gradient (NPG). NPG utilizes a non-stationary reward function, and encourages a continuous adaptation of the policy to capture the entire Pareto front efficiently. We introduce two novel reward functions with elements from the dominant paradigms of scalarization and evolution. To handle non-stationarity, we propose a new exploration scheme using cosine temperature decay with warm restarts. For fast and accurate architecture evaluation, we introduce a novel pre-trained shared model that we continuously fine-tune throughout training. Our extensive experimental study with various datasets shows that our framework can approximate the full Pareto front well at fast speeds. Moreover, our discovered cells can achieve supreme predictive performance compared to other multi-objective NAS methods, and other single-objective NAS methods at similar network sizes. Our work demonstrates the potential of NPG as a simple, efficient, and effective paradigm for multi-objective NAS.

Motivation & Objective

• To address the challenge of efficiently and accurately approximating the full Pareto front in multi-objective neural architecture search (NAS).
• To overcome limitations of scalarization and evolutionary methods, which require multiple runs or suffer from high search cost and bias.
• To develop a unified RL-based framework that dynamically adapts policies to explore diverse, Pareto-optimal architectures in a single training run.
• To improve architecture evaluation speed and accuracy using a continuously fine-tuned pre-trained shared model.

Proposed method

• Introduces two novel non-stationary reward functions: ADF (target-based desirability) for gradual scalarization and ADC (Pareto dominance-based) for band-wise expansion of the Pareto front.
• Employs a non-stationary policy gradient framework where the reward function evolves over time to guide continuous policy adaptation.
• Applies cosine temperature decay with warm restarts to balance exploration and exploitation during non-stationary training.
• Uses a pre-trained shared model that is continuously fine-tuned upon each architecture sample to enable fast and accurate validation accuracy estimation.
• Leverages a controller policy to generate architectures in a cell-based search space, with rewards based on Pareto dominance or scalarized objectives.
• Enables end-to-end training where the policy learns to generate architectures that progressively dominate or approach the true Pareto front.

Experimental results

Research questions

• RQ1 Can a non-stationary policy gradient framework effectively approximate the full Pareto front in multi-objective NAS with a single training run?
• RQ2 How does the proposed ADF and ADC reward functions compare to traditional scalarization and evolutionary methods in terms of Pareto front coverage and diversity?
• RQ3 To what extent does the cosine temperature decay with warm restarts improve exploration in non-stationary RL for NAS?
• RQ4 How accurate and fast is architecture evaluation using the continuously fine-tuned pre-trained shared model compared to training from scratch?
• RQ5 Can architectures discovered via NPG-NAS generalize well to downstream tasks like ImageNet transfer learning?

Key findings

• ADF-L achieved 83.99% accuracy on CIFAR-100 with only 5.73 million parameters, outperforming DenseNet-BC by 77.61% fewer parameters.
• ADC-L10 and ADF-L100 achieved higher top-1 accuracy on ImageNet than DPP-Net-PNAS while using only 1/10 of its parameters.
• NPG-NAS transferred cells from CIFAR-10 and CIFAR-100 to ImageNet, achieving 76.67% top-1 accuracy with 7.89 million parameters, surpassing most NAS methods at similar sizes.
• The framework achieved a more uniform and extended Pareto front than random search (RS), with better range and diversity, especially for ADC.
• The pre-trained shared model enabled fast convergence in just 50 steps and showed high correlation (implied by strong validation accuracy) between search-time and stand-alone training accuracy.
• On three-objective CIFAR-10, ADF and ADC outperformed RS and performed comparably to M-DF, with ADC showing broader coverage on the right side of the Pareto front.