[Paper Review] NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm

Zhichao Lu, Ian Whalen+5 more|arXiv (Cornell University)|Oct 8, 2018

Machine Learning and Data Classification47 references73 citations

TL;DR

NSGA-Net uses a multi-objective genetic algorithm with crossover and Bayesian-network exploitation to search neural architectures that trade off accuracy and FLOPs, yielding a Pareto frontier with competitive CIFAR-10 results and substantially lower compute.

ABSTRACT

This paper introduces NSGA-Net -- an evolutionary approach for neural architecture search (NAS). NSGA-Net is designed with three goals in mind: (1) a procedure considering multiple and conflicting objectives, (2) an efficient procedure balancing exploration and exploitation of the space of potential neural network architectures, and (3) a procedure finding a diverse set of trade-off network architectures achieved in a single run. NSGA-Net is a population-based search algorithm that explores a space of potential neural network architectures in three steps, namely, a population initialization step that is based on prior-knowledge from hand-crafted architectures, an exploration step comprising crossover and mutation of architectures, and finally an exploitation step that utilizes the hidden useful knowledge stored in the entire history of evaluated neural architectures in the form of a Bayesian Network. Experimental results suggest that combining the dual objectives of minimizing an error metric and computational complexity, as measured by FLOPs, allows NSGA-Net to find competitive neural architectures. Moreover, NSGA-Net achieves error rate on the CIFAR-10 dataset on par with other state-of-the-art NAS methods while using orders of magnitude less computational resources. These results are encouraging and shows the promise to further use of EC methods in various deep-learning paradigms.

Motivation & Objective

• Automate neural architecture search (NAS) while explicitly optimizing multiple objectives.
• Balance predictive performance with compute-related constraints (FLOPs).
• Explore architectures beyond repeated blocks by searching across full network structures.
• Leverage history through Bayesian learning to guide efficient search.

Proposed method

• Represent architectures as bit-string encodings of phase-level blocks (operations per phase).
• Use NSGA-II to maintain a diverse Pareto-front across error and FLOPs objectives.
• Apply crossover to recombine architectural blocks and mutation to introduce diversity.
• Exploit search history via a Bayesian Network to sample promising offspring (BOA-inspired).
• Two-stage search: exploration (crossover/mutation) followed by exploitation (BN-guided sampling).
• Train candidate architectures with SGD and cosine annealing to evaluate validation error during search.

Experimental results

Research questions

• RQ1 Can a multi-objective EA locate a Pareto frontier of neural architectures balancing accuracy and computational cost?
• RQ2 Does incorporating crossover and Bayesian-network exploitation improve search efficiency and quality over random or single-objective methods?
• RQ3 How transferable are NSGA-Net found architectures to related datasets (e.g., CIFAR-100) or settings?
• RQ4 How does NSGA-Net compare to state-of-the-art NAS methods in accuracy, parameter count, and compute?
• RQ5 What architectural trends emerge when optimizing for multiple objectives?

Key findings

• NSGA-Net discovers a Pareto-front of architectures balancing error and FLOPs on CIFAR-10.
• The best-found architecture achieves 3.85% test error with 3.34M parameters and 1290 MFLOPs.
• NSGA-Net requires substantially less compute (8 GPU-days) than some competing NAS methods while delivering competitive accuracy.
• A Bayesian-network-based exploitation stage and crossover improve frontier quality versus random search or mutation-only EA.
• Found architectures on CIFAR-10 transfer reasonably to CIFAR-100, achieving competitive accuracy with fewer parameters.
• Ablation studies show crossover and BN-based offspring creation both contribute to better trade-offs.