[Paper Review] Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution

Thomas Elsken, Jan Hendrik Metzen+1 more|arXiv (Cornell University)|Apr 24, 2018

Evolutionary Algorithms and ApplicationsComputer Science214 citations

TL;DR

LEMONADE is an evolutionary, multi-objective neural architecture search method that uses Lamarckian inheritance via network morphisms and approximate network morphisms to efficiently approximate the Pareto front across accuracy and resource constraints.

ABSTRACT

Neural Architecture Search aims at automatically finding neural architectures that are competitive with architectures designed by human experts. While recent approaches have achieved state-of-the-art predictive performance for image recognition, they are problematic under resource constraints for two reasons: (1)the neural architectures found are solely optimized for high predictive performance, without penalizing excessive resource consumption, (2) most architecture search methods require vast computational resources. We address the first shortcoming by proposing LEMONADE, an evolutionary algorithm for multi-objective architecture search that allows approximating the entire Pareto-front of architectures under multiple objectives, such as predictive performance and number of parameters, in a single run of the method. We address the second shortcoming by proposing a Lamarckian inheritance mechanism for LEMONADE which generates children networks that are warmstarted with the predictive performance of their trained parents. This is accomplished by using (approximate) network morphism operators for generating children. The combination of these two contributions allows finding models that are on par or even outperform both hand-crafted as well as automatically-designed networks.

Motivation & Objective

• Motivate automatic neural architecture search under multiple resource and accuracy objectives.
• Propose an evolutionary framework that returns the Pareto front rather than a single solution.
• Reduce computation by using function-preserving and approximate operators to warm-start children.
• Enable handling of arbitrary search spaces including complex topologies and skip connections.

Proposed method

• Use network morphisms to create function-preserving child networks that retain trained performance (Net2DeeperNet, Net2WiderNet concepts).
• Introduce approximate network morphisms (ANMs) to also allow shrinking networks while preserving performance via knowledge distillation.
• Develop LEMONADE, an evolutionary algorithm that maintains a population on the Pareto front and uses cheap objectives to bias parent/child selection before expensive evaluations.
• Mutate networks with NM and ANM operators to generate offspring and apply Lamarckian inheritance by initializing offspring with weights informed by parents.
• Differentiate evaluation: optimize cheap objectives (parameters, FLOPs, inference time) with high-frequency sampling and expensive objectives (validation accuracy) less frequently to build a diverse Pareto front.
• Return a set of architectures on the Pareto front rather than a single trade-off, and support arbitrary search spaces including cells and full architectures.

Experimental results

Research questions

• RQ1 How can multi-objective NAS efficiently approximate the Pareto front across accuracy and resource metrics?
• RQ2 Can Lamarckian inheritance via network morphisms speed up NAS by warm-starting offspring?
• RQ3 How well does LEMONADE perform on unconstrained search spaces and across different objectives (e.g., parameters, FLOPs, inference time) compared to hand-designed or other NAS methods?

Key findings

• LEMONADE discovers architectures competitive with MobileNetV2 and NASNet under multiple objectives within 5 days on 16 GPUs.
• The search yields a Pareto front spanning 10,000 to 10,000,000 parameters and shows improved inference time over baselines.
• On CIFAR-10, LEMONADE outperforms NASNet and MobileNetV2 in several resource-conscious regimes and is competitive with or superior to state-of-the-art multi-objective methods.
• Using 80 GPU days, LEMONADE achieves competitive results versus methods like Zoph et al. (20014) which used much more compute.
• Transferred cells to ImageNet64x64 and ImageNet mobile settings, cells dominated baselines such as NASNets and MobileNets V2 in transfer experiments.
• Table 1 shows LEMONADE matching or surpassing several NAS methods across varying model sizes under identical training conditions.