[Paper Review] Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessment

Zhichao Lu, Ran Cheng+3 more|arXiv (Cornell University)|Aug 8, 2022

Advanced Multi-Objective Optimization Algorithms4 citations

TL;DR

This paper introduces EvoXBench, a GPU-free, end-to-end pipeline for generating standardized benchmark test problems for evolutionary multiobjective optimization (EMO) in neural architecture search (NAS). It formulates NAS as a general multiobjective optimization problem, enabling efficient evaluation of EMO algorithms across diverse datasets, search spaces, and hardware platforms, with empirical validation using six algorithms on up to eight objectives.

ABSTRACT

The ongoing advancements in network architecture design have led to remarkable achievements in deep learning across various challenging computer vision tasks. Meanwhile, the development of neural architecture search (NAS) has provided promising approaches to automating the design of network architectures for lower prediction error. Recently, the emerging application scenarios of deep learning have raised higher demands for network architectures considering multiple design criteria: number of parameters/floating-point operations, and inference latency, among others. From an optimization point of view, the NAS tasks involving multiple design criteria are intrinsically multiobjective optimization problems; hence, it is reasonable to adopt evolutionary multiobjective optimization (EMO) algorithms for tackling them. Nonetheless, there is still a clear gap confining the related research along this pathway: on the one hand, there is a lack of a general problem formulation of NAS tasks from an optimization point of view; on the other hand, there are challenges in conducting benchmark assessments of EMO algorithms on NAS tasks. To bridge the gap: (i) we formulate NAS tasks into general multi-objective optimization problems and analyze the complex characteristics from an optimization point of view; (ii) we present an end-to-end pipeline, dubbed $ exttt{EvoXBench}$, to generate benchmark test problems for EMO algorithms to run efficiently -- without the requirement of GPUs or Pytorch/Tensorflow; (iii) we instantiate two test suites comprehensively covering two datasets, seven search spaces, and three hardware devices, involving up to eight objectives. Based on the above, we validate the proposed test suites using six representative EMO algorithms and provide some empirical analyses. The code of $ exttt{EvoXBench}$ is available from $\href{https://github.com/EMI-Group/EvoXBench}{ m{here}}$.

Motivation & Objective

• Address the lack of a general multiobjective optimization formulation for NAS from an optimization perspective.
• Overcome the challenge of comparing EMO algorithms on NAS tasks due to inconsistent search spaces, evaluation procedures, and computational requirements.
• Develop a scalable, efficient benchmarking pipeline that eliminates dependency on deep learning frameworks and GPUs.
• Provide comprehensive test suites covering multiple datasets, search spaces, and hardware platforms for rigorous EMO algorithm evaluation.
• Enable empirical analysis of EMO performance across trade-offs between prediction error, model complexity, and hardware efficiency (e.g., latency, energy).

Proposed method

• Formulate NAS as a general multiobjective optimization problem (MOP) with objectives including prediction error, model complexity (parameters, FLOPs), and hardware efficiency (latency, energy).
• Design EvoXBench as an end-to-end pipeline that generates benchmark instances without requiring PyTorch, TensorFlow, or GPU access.
• Implement a modular framework to support multiple datasets (e.g., CIFAR-10, ImageNet), search spaces (e.g., MobileNet, ResNet, Transformer), and hardware devices (e.g., GPU, Eyeriss).
• Define up to eight objectives per test instance, including both model-level (e.g., accuracy, FLOPs) and hardware-level (e.g., inference latency, energy) metrics.
• Integrate EMO algorithms (e.g., NSGA-II) into the pipeline for performance evaluation using metrics like hypervolume (HV) and convergence analysis.
• Visualize Pareto-optimal architectures across trade-off subsets to analyze architectural patterns and operator preferences under different objectives.

(a) Standard NAS Pipeline

Experimental results

Research questions

• RQ1 How can NAS be systematically formulated as a general multiobjective optimization problem with well-defined objectives and constraints?
• RQ2 To what extent can EMO algorithms effectively explore the trade-offs between prediction error, model complexity, and hardware efficiency in NAS?
• RQ3 How do architectural choices (e.g., convolution types, pooling operations) vary across different objective trade-off regions in the Pareto front?
• RQ4 What are the consistent design patterns in Pareto-optimal architectures across different search spaces and hardware platforms?
• RQ5 Can a GPU-free, framework-agnostic benchmarking pipeline like EvoXBench enable fair and efficient evaluation of EMO algorithms in NAS?

Key findings

• The $3\times3$ convolution is consistently preferred across all trade-off scenarios, indicating its strong positive impact on prediction error.
• The $3\times3$ average pooling operation is rarely used in Pareto-optimal architectures, suggesting it negatively affects prediction performance.
• A transition from accuracy-complexity to accuracy-latency trade-offs can be achieved by systematically removing $1\times1$ convolutions.
• Inference latency and energy consumption are strongly correlated on Eyeriss, indicating that faster models tend to be more energy-efficient.
• Trade-offs exist between energy efficiency and arithmetic intensity, as shown by non-parallel, crisscrossing objective relationships in the multiobjective landscape.
• EvoXBench enables efficient, reproducible, and fair evaluation of EMO algorithms without GPU or deep learning framework dependencies, validating its utility for NAS benchmarking.

(b) Our NAS Benchmark tailored for EMO