[Paper Review] Evolving Deep Neural Networks by Multi-objective Particle Swarm Optimization for Image Classification

Bin Wang, Yanan Sun+2 more|arXiv (Cornell University)|Mar 21, 2019

Advanced Neural Network Applications22 references4 citations

TL;DR

This paper proposes MOCNN, a multi-objective particle swarm optimization framework that evolves deep convolutional neural networks (CNNs) by simultaneously optimizing classification accuracy and floating-point operations (FLOPs) to balance performance and efficiency. It introduces a novel encoding strategy for DenseNet-based architectures, employs OMOPSO for Pareto-optimal solution exploration, and develops a distributed GPU infrastructure enabling near-linear speedup, achieving 95.51% accuracy on CIFAR-10—surpassing DenseNet-121 while reducing computational cost.

ABSTRACT

In recent years, convolutional neural networks (CNNs) have become deeper in order to achieve better classification accuracy in image classification. However, it is difficult to deploy the state-of-the-art deep CNNs for industrial use due to the difficulty of manually fine-tuning the hyperparameters and the trade-off between classification accuracy and computational cost. This paper proposes a novel multi-objective optimization method for evolving state-of-the-art deep CNNs in real-life applications, which automatically evolves the non-dominant solutions at the Pareto front. Three major contributions are made: Firstly, a new encoding strategy is designed to encode one of the best state-of-the-art CNNs; With the classification accuracy and the number of floating point operations as the two objectives, a multi-objective particle swarm optimization method is developed to evolve the non-dominant solutions; Last but not least, a new infrastructure is designed to boost the experiments by concurrently running the experiments on multiple GPUs across multiple machines, and a Python library is developed and released to manage the infrastructure. The experimental results demonstrate that the non-dominant solutions found by the proposed algorithm form a clear Pareto front, and the proposed infrastructure is able to almost linearly reduce the running time.

Motivation & Objective

• To address the challenge of deploying state-of-the-art deep CNNs in industrial applications due to high hyperparameter tuning complexity and trade-offs between accuracy and inference latency.
• To develop a multi-objective optimization method that simultaneously improves classification accuracy and reduces computational cost (measured by FLOPs) for real-world deployment.
• To design an efficient, scalable infrastructure leveraging multiple GPUs across machines to drastically reduce training time for evolutionary CNN search.
• To generate a set of non-dominated, Pareto-optimal CNN architectures that allow industrial users to select models based on their specific accuracy and latency requirements.

Proposed method

• A new encoding strategy is designed to represent DenseNet-based CNNs by parameterizing key hyperparameters: number of dense blocks, growth rate, and number of layers per block.
• The method formulates a multi-objective optimization problem with two objectives: maximizing classification accuracy and minimizing FLOPs (a proxy for inference latency).
• A modified OMOPSO algorithm is employed to evolve a population of CNN configurations, maintaining diversity and convergence toward the Pareto front.
• A distributed computing infrastructure is implemented using multiple GPUs across multiple machines, with a custom Python library to manage concurrent training and evaluation of individuals.
• The evolutionary process uses full training with Adam optimizer on full datasets, ensuring reliable fitness evaluation despite high computational cost.
• The framework is evaluated on CIFAR-10 using standard data augmentation and training protocols to ensure fair comparison with baseline models like DenseNet-121.

Experimental results

Research questions

• RQ1 Can multi-objective particle swarm optimization effectively evolve deep CNNs that balance high classification accuracy and low computational cost?
• RQ2 How does the proposed encoding strategy enable efficient exploration of the hyperparameter space for DenseNet-based architectures?
• RQ3 To what extent can a distributed GPU infrastructure reduce the runtime of evolutionary CNN search while maintaining solution quality?
• RQ4 Does the evolved CNN architecture outperform standard models like DenseNet-121 in terms of both accuracy and FLOPs within the defined search space?
• RQ5 Can the method generate a diverse set of non-dominated solutions that form a clear Pareto front for industrial deployment?

Key findings

• The proposed MOCNN framework successfully generated a clear Pareto front of non-dominated CNN solutions, demonstrating a balanced trade-off between classification accuracy and FLOPs.
• The optimized MOCNN achieved 95.51% test accuracy on CIFAR-10, outperforming DenseNet-121 (94.77%) within the same search space.
• The model with 50 individuals in the swarm provided broader coverage of the Pareto front than the 20-individual run, despite slower convergence.
• The distributed GPU infrastructure reduced the total runtime from ~3 weeks (single GPU) to approximately 3 days (8–10 GPUs), achieving near-linear speedup.
• The method demonstrated that evolutionary search can yield models competitive with or better than hand-crafted architectures like DenseNet-121, even under computational constraints.