[Paper Review] Efficient Hyperparameter Optimization in Deep Learning Using a Variable Length Genetic Algorithm

Xueli Xiao, Ming Yan+3 more|arXiv (Cornell University)|Jun 23, 2020

Machine Learning and Data Classification25 references80 citations

TL;DR

The paper introduces a variable length genetic algorithm (GA) for CNN hyperparameter optimization that grows model depth over phases, uses crossover and weight transfer, and demonstrates superior performance vs fixed-length GA, random search, and large-scale evolution under time/resource constraints.

ABSTRACT

Convolutional Neural Networks (CNN) have gained great success in many artificial intelligence tasks. However, finding a good set of hyperparameters for a CNN remains a challenging task. It usually takes an expert with deep knowledge, and trials and errors. Genetic algorithms have been used in hyperparameter optimizations. However, traditional genetic algorithms with fixed-length chromosomes may not be a good fit for optimizing deep learning hyperparameters, because deep learning models have variable number of hyperparameters depending on the model depth. As the depth increases, the number of hyperparameters grows exponentially, and searching becomes exponentially harder. It is important to have an efficient algorithm that can find a good model in reasonable time. In this article, we propose to use a variable length genetic algorithm (GA) to systematically and automatically tune the hyperparameters of a CNN to improve its performance. Experimental results show that our algorithm can find good CNN hyperparameters efficiently. It is clear from our experiments that if more time is spent on optimizing the hyperparameters, better results could be achieved. Theoretically, if we had unlimited time and CPU power, we could find the optimized hyperparameters and achieve the best results in the future.

Motivation & Objective

• Motivate the challenge of CNN hyperparameter optimization and the need for efficient search when model depth varies.
• Propose a variable length GA that progressively grows CNN depth to search a larger, non-fixed hyperparameter space.
• Demonstrate that the method finds high-performing CNN configurations within practical compute budgets.
• Compare against baseline methods (random search, fixed-length GA, large-scale evolution) under similar time/resource constraints.

Proposed method

• Use a variable length GA that starts with two convolutional layers and gradually adds layers across phases.
• Encode hyperparameters in a chromosome that expands with each phase, propagating best phase solutions to deeper models.
• Incorporate crossover and mutation to generate new hyperparameter configurations, with weight transfer from the previous best model when expanding depth.
• Evaluate fitness as validation accuracy after a small, fixed number of epochs (five) to save compute, with weight transfer to initialize deeper models.
• Train the best model from each phase further to convergence after stopping criteria are met.
• Allow the search to stop when adding layers no longer yields improvement beyond a 0.01 drop in best fitness across phases.

Experimental results

Research questions

• RQ1 Can a variable length chromosome GA effectively optimize CNN hyperparameters without a predefined maximum depth?
• RQ2 Does crossover plus weight transfer enable more efficient discovery of high-performing CNN architectures under limited computational resources?
• RQ3 How does the variable length GA perform compared to random search, fixed-length GA, and large-scale evolution under similar time budgets?
• RQ4 What stopping conditions reliably indicate when deeper architectures stop providing meaningful gains?

Key findings

• The variable length GA discovered an 88.92% test accuracy model on CIFAR-10 within ~24.55 hours.
• The method shows rapid early gains when increasing depth, with improvements tapering in later phases.
• Under similar time constraints, the proposed method outperformed random search (58.66%), classical GA (80.75%), and large-scale evolution (51.90%).
• The overall evolution completed in under 25 hours, and a best model from Phase 13 was selected for further training.
• Weight transfer during depth expansion saved training time and helped evaluate deeper architectures more efficiently.