[Paper Review] Recombination of Artificial Neural Networks

Aaron Vose, Jacob Balma+6 more|arXiv (Cornell University)|Jan 12, 2019

Machine Learning and Data Classification22 references4 citations

TL;DR

This paper proposes a novel genetic algorithm (GA) for hyperparameter optimization in deep neural networks that introduces tri-parental reproduction—two parents contributing hyperparameters and one contributing model parameters—alongside chromosomal crossover and decoupling of parameters from hyperparameters. The method accelerates convergence and improves final accuracy across diverse architectures like CNNs, RNNs, and capsule networks by combining PBT with GA, enabling faster adaptation and better elimination of deleterious hyperparameter sets.

ABSTRACT

We propose a genetic algorithm (GA) for hyperparameter optimization of artificial neural networks which includes chromosomal crossover as well as a decoupling of parameters (i.e., weights and biases) from hyperparameters (e.g., learning rate, weight decay, and dropout) during sexual reproduction. Children are produced from three parents; two contributing hyperparameters and one contributing the parameters. Our version of population-based training (PBT) combines traditional gradient-based approaches such as stochastic gradient descent (SGD) with our GA to optimize both parameters and hyperparameters across SGD epochs. Our improvements over traditional PBT provide an increased speed of adaptation and a greater ability to shed deleterious genes from the population. Our methods improve final accuracy as well as time to fixed accuracy on a wide range of deep neural network architectures including convolutional neural networks, recurrent neural networks, dense neural networks, and capsule networks.

Motivation & Objective

• To address the challenge of hyperparameter optimization (HPO) in deep neural networks, where manual tuning is impractical due to vast search spaces.
• To improve upon existing population-based training (PBT) by incorporating sexual reproduction with crossover and tri-parental inheritance.
• To enhance search efficiency and convergence speed by enabling the shedding of deleterious hyperparameter configurations and faster adaptation to optimal regimes.
• To extend PBT into a full-fledged genetic algorithm with biologically inspired features such as demes and ploidy for better exploration of the hyperparameter space.

Proposed method

• The method employs a genetic algorithm that decouples hyperparameters (e.g., learning rate, weight decay) from model parameters (weights and biases), treating them as distinct genetic components.
• Children are generated from three parents: two contributing hyperparameters via crossover, and one contributing trained model parameters.
• Chromosomal crossover is applied to hyperparameters only, enabling recombination of beneficial gene combinations while preserving the integrity of trained network weights.
• The algorithm integrates with stochastic gradient descent (SGD) by running GA generations in parallel with SGD training epochs, allowing dynamic adaptation of hyperparameters during training.
• Demes are introduced as isolated subpopulations with limited migration to prevent premature convergence and improve coverage of the hyperparameter space.
• Ploidy is explored in the context of neural network ensembles, where multiple subnetworks from different parents may enable sexual reproduction of parameters, though current implementation uses haploid parameter inheritance.

Experimental results

Research questions

• RQ1 Can a genetic algorithm with tri-parental reproduction and crossover improve hyperparameter optimization in deep neural networks compared to traditional PBT?
• RQ2 How does decoupling hyperparameters from model parameters affect the efficiency and effectiveness of hyperparameter search?
• RQ3 Can the use of demes and migration strategies improve exploration and reduce convergence to local optima in HPO?
• RQ4 To what extent can crossover of hyperparameters accelerate convergence and improve final model accuracy across diverse architectures?
• RQ5 Can the integration of PBT with a full-fledged genetic algorithm lead to faster time-to-fixed-accuracy and higher final accuracy?

Key findings

• The proposed GA improves final accuracy across a wide range of deep neural network architectures, including convolutional, recurrent, dense, and capsule networks, compared to default hyperparameter settings.
• The method reduces time to reach fixed accuracy by enabling faster adaptation through effective recombination and pruning of deleterious hyperparameter sets.
• Incorporating crossover and tri-parental inheritance allows for more efficient exploration of the hyperparameter space, leading to better convergence behavior than standard PBT.
• The use of demes with controlled migration enhances population diversity and reduces the risk of premature convergence to suboptimal solutions.
• The approach demonstrates improved performance over baseline PBT by combining gradient-based training with evolutionary search, particularly in dynamic hyperparameter scheduling.
• Preliminary exploration of ploidy via ensembles suggests potential for future extension to enable crossover of model parameters, though current implementation treats parameters as haploid and non-recombining.