[Paper Review] Automatically Evolving CNN Architectures Based on Blocks

Yanan Sun, Bing Xue+2 more|arXiv (Cornell University)|Oct 28, 2018

Advanced Neural Network ApplicationsComputer Science25 references22 citations

TL;DR

This paper proposes an automatic genetic algorithm, AE-CNN, that evolves CNN architectures using ResNet and DenseNet blocks with variable-length encoding, enabling both local and global search via novel crossover and mutation operators. It achieves state-of-the-art accuracy on CIFAR-10 and CIFAR-100 with significantly lower computational cost than peer methods.

ABSTRACT

The performance of Convolutional Neural Networks (CNNs) highly relies on their architectures. In order to design a CNN with promising performance, extended expertise in both CNNs and the investigated problem is required, which is not necessarily held by every user interested in CNNs or the problem domain. In this paper, we propose to automatically evolve CNN architectures by using a genetic algorithm based on ResNet blocks and DenseNet blocks. The proposed algorithm is extbf{completely} automatic in designing CNN architectures, particularly, neither pre-processing before it starts nor post-processing on the designed CNN is needed. Furthermore, the proposed algorithm does not require users with domain knowledge on CNNs, the investigated problem or even genetic algorithms. The proposed algorithm is evaluated on CIFAR10 and CIFAR100 against 18 state-of-the-art peer competitors. Experimental results show that it outperforms state-of-the-art CNNs hand-crafted and CNNs designed by automatic peer competitors in terms of the classification accuracy, and achieves the competitive classification accuracy against semi-automatic peer competitors. In addition, the proposed algorithm consumes much less time than most peer competitors in finding the best CNN architectures.

Motivation & Objective

• To develop a completely automatic CNN architecture search method that requires no expert knowledge in CNNs, the problem domain, or genetic algorithms.
• To overcome limitations of existing automatic and semi-automatic CNN architecture search methods, including high computational cost and reliance on human-designed components.
• To design a genetic algorithm with variable-length representation that can adaptively evolve optimal network depth and block composition.
• To enable effective global and local search through a novel crossover operator and tailored mutation operators for block-based CNNs.
• To achieve high classification accuracy on image datasets while minimizing GPU resource consumption.

Proposed method

• The method uses a variable-length encoding scheme to represent CNN architectures composed of ResNet and DenseNet blocks, allowing dynamic depth evolution.
• A novel crossover operator is designed to handle variable-length individuals by aligning and recombining block sequences based on structural similarity.
• Multiple mutation operators are applied to blocks, including block insertion, deletion, replacement, and hyperparameter mutation, enabling both local and global search.
• The algorithm employs a fitness function based on validation accuracy, with no need for pre-processing or post-processing steps.
• Evolution is driven by generational selection, where fitter architectures are selected for reproduction over multiple generations.
• The entire process is fully automated, requiring no human intervention or domain-specific tuning.

Experimental results

Research questions

• RQ1 Can a genetic algorithm based on ResNet and DenseNet blocks automatically evolve high-performing CNN architectures without human expertise?
• RQ2 Does the proposed variable-length encoding with a novel crossover operator improve search efficiency and performance compared to fixed-length or non-crossover methods?
• RQ3 How does the performance of the proposed automatic CNN architecture search method compare to state-of-the-art hand-crafted and automated peer methods in terms of accuracy and computational cost?
• RQ4 Can the method achieve competitive or superior accuracy on CIFAR-10 and CIFAR-100 while consuming significantly fewer GPU days than existing automatic methods?

Key findings

• AE-CNN achieved a test error of 4.7% on CIFAR-10, outperforming all state-of-the-art hand-crafted CNNs, including ResNet (depth=1,202) and DenseNet (k=12), by 0.5% to 4.5%.
• On CIFAR-100, AE-CNN achieved lower error than Maxout, Network in Network, Highway Network, and All-CNN, and was comparable to FractalNet and ResNet (depth=101).
• AE-CNN consumed only 1/5 the GPU days of Hierarchical Evolution and 1/3 the GPU days of Block-QNN-S on CIFAR-10 and CIFAR-100, respectively, despite superior performance.
• Among fully automatic methods, AE-CNN achieved the best accuracy with 4.7% error on CIFAR-10, outperforming NAS (5.4% error), Large-scale Evolution (5.4% error), and MetaQNN.
• The evolved CNN had 32M parameters, fewer than VGG and FractalNet, and comparable to ResNet (depth=1,202), indicating high parameter efficiency.
• AE-CNN demonstrated competitive performance against semi-automatic methods like EAS and Block-QNN-S, while requiring no human expertise or pre-defined base networks.