[Paper Review] Multi-objective Architecture Search for CNNs.

Thomas Elsken, Jan Hendrik Metzen+1 more|arXiv (Cornell University)|Apr 24, 2018

Advanced Neural Network Applications28 references25 citations

TL;DR

This paper proposes NASH and Pareto-NASH for efficient, multi-objective neural architecture search in CNNs. NASH reduces training cost via network morphisms and aggressive learning rate scheduling, achieving <4% error on CIFAR10 in 3 days. Pareto-NASH enables single-run discovery of Pareto-optimal architectures across accuracy and parameter count, finding models with 3.5% error and 4M parameters, or 4.6% error and <1M parameters within 56 GPU days.

ABSTRACT

Architecture search aims at automatically finding neural architectures that are competitive with architectures designed by human experts. While recent approaches have come close to matching the predictive performance of manually designed architectures for image recognition, these approaches are problematic under constrained resources for two reasons: first, the architecture search itself requires vast computational resources for most proposed methods. Secondly, the found neural architectures are solely optimized for high predictive performance without penalizing excessive resource consumption. We address the first shortcoming by proposing NASH, an architecture search which considerable reduces the computational resources required for training novel architectures by applying network morphisms and aggressive learning rate schedules. On CIFAR10, NASH finds architectures with errors below 4% in only 3 days. We address the second shortcoming by proposing Pareto-NASH, a method for multi-objective architecture search that allows approximating the Pareto-front of architectures under multiple objective, such as predictive performance and number of parameters, in a single run of the method. Within 56 GPU days of architecture search, Pareto-NASH finds a model with 4M parameters and test error of 3.5%, as well as a model with less than 1M parameters and test error of 4.6%.

Motivation & Objective

• To reduce the high computational cost of neural architecture search (NAS) for CNNs.
• To address the lack of efficiency-awareness in NAS by optimizing for both accuracy and model size.
• To enable simultaneous discovery of architectures that balance predictive performance and parameter count.
• To develop a method that scales efficiently under resource constraints.

Proposed method

• NASH employs network morphisms to generate and train new architectures efficiently from a parent network.
• It uses aggressive learning rate schedules to accelerate convergence during architecture search.
• Pareto-NASH extends NASH to multi-objective optimization by maintaining a diverse set of architectures across accuracy and parameter count.
• The method uses a single search run to approximate the Pareto front of trade-offs between performance and model size.
• Architectures are evaluated and updated based on multiple objectives using a differentiable search strategy.
• Search efficiency is achieved by reusing features and gradients across architectures via morphism-based parameter sharing.

Experimental results

Research questions

• RQ1 Can architecture search be made significantly more efficient without sacrificing performance?
• RQ2 Can a single search run identify a diverse set of architectures that represent trade-offs between accuracy and model size?
• RQ3 How much computational cost can be reduced while maintaining competitive accuracy in NAS?
• RQ4 Can aggressive learning rate scheduling improve convergence speed in architecture search?
• RQ5 What is the achievable trade-off between model accuracy and parameter count using a multi-objective NAS approach?

Key findings

• NASH achieves test error below 4% on CIFAR10 in only 3 days of training, significantly reducing search time.
• Pareto-NASH discovers a model with 4 million parameters and 3.5% test error within 56 GPU days.
• A smaller model with less than 1 million parameters and 4.6% test error is also found using Pareto-NASH.
• The method achieves competitive performance with substantially reduced computational cost compared to prior NAS approaches.
• The Pareto front approximation includes multiple high-performing, efficient architectures, demonstrating effective trade-off discovery.
• The use of network morphisms and aggressive learning rate scheduling enables faster convergence and lower resource usage during search.