[Paper Review] Large-Scale Evolution of Image Classifiers

Esteban Real, Sherry Moore+6 more|arXiv (Cornell University)|Mar 3, 2017

Evolutionary Algorithms and ApplicationsComputer Science39 references544 citations

TL;DR

The paper demonstrates that large-scale neuro-evolution starting from simple, non-convolutional beginnings can yield fully trained image classifiers that rival hand-designed models on CIFAR-10 and CIFAR-100, achieving 94.6% test accuracy on CIFAR-10 (95.6% with ensembling) and 77.0% on CIFAR-100.

ABSTRACT

Neural networks have proven effective at solving difficult problems but designing their architectures can be challenging, even for image classification problems alone. Our goal is to minimize human participation, so we employ evolutionary algorithms to discover such networks automatically. Despite significant computational requirements, we show that it is now possible to evolve models with accuracies within the range of those published in the last year. Specifically, we employ simple evolutionary techniques at unprecedented scales to discover models for the CIFAR-10 and CIFAR-100 datasets, starting from trivial initial conditions and reaching accuracies of 94.6% (95.6% for ensemble) and 77.0%, respectively. To do this, we use novel and intuitive mutation operators that navigate large search spaces; we stress that no human participation is required once evolution starts and that the output is a fully-trained model. Throughout this work, we place special emphasis on the repeatability of results, the variability in the outcomes and the computational requirements.

Motivation & Objective

• Motivate automated architecture discovery for image classification to minimize human intervention.
• Show that simple evolutionary processes can yield high-performing convolutional networks at scale.
• Demonstrate repeatability, variability, and computational requirements of large-scale neuro-evolution.

Proposed method

• Use a massively parallel, asynchronous evolutionary algorithm with tournament selection over a population of 1000 neural architectures.
• Encode architectures as a mutable graph (DNA) of activation tensors connected by convolutional edges.
• Apply mutations that insert/remove convolutions, alter strides, channels, filter sizes, and add/remove skip connections, among others.
• Train each candidate for a fixed number of steps (25,600) with SGD and momentum; evaluate fitness via validation accuracy.
• Weight inheritance allows mutations to reuse weights when layer shapes match, enabling fully trained offspring without starting from scratch.

Experimental results

Research questions

• RQ1 Can a simple, one-shot evolutionary process starting from trivial initial conditions produce fully trained image classifiers rivaling hand-designed models?
• RQ2 How does population size and per-individual training steps affect performance, repeatability, and convergence to local optima in large-scale neuro-evolution?
• RQ3 Is the CIFAR-10 evolutionizable without changes to the algorithm when applied to CIFAR-100?
• RQ4 What is the computational cost and scalability of the approach, and how does weight inheritance impact results?
• RQ5 What is the role of mutations versus recombination in escaping local optima and improving architectures?

Key findings

• Evolved models reach CIFAR-10 test accuracies around 94.1% on average with 94.6% for the top model and 95.6% with ensemble.
• On CIFAR-100, the evolved single experiment achieves 77.0% test accuracy.
• Across five large-scale runs, the best model by validation accuracy achieved 94.6% with a standard deviation of 0.4%.
• The total computation across five experiments was about 4×10^20 FLOPs, with ~9×10^19 FLOPs per experiment.
• Weight inheritance is important for achieving high accuracy; disabling it reduces performance.
• Increasing population size and training steps per individual improves results and helps escape local optima.