[Paper Review] Survey on Evolutionary Deep Learning: Principles, Algorithms, Applications and Open Issues

Nan Li, Lianbo Ma+4 more|arXiv (Cornell University)|Aug 23, 2022

Evolutionary Algorithms and Applications4 citations

TL;DR

This survey presents a comprehensive analysis of evolutionary deep learning (EDL) as an automated machine learning (AutoML) framework, integrating evolutionary computation (EC) to optimize deep learning pipelines—from feature engineering and model generation to deployment. It introduces a novel taxonomy for EDL, details solution representations and search paradigms, and identifies key challenges and future research directions.

ABSTRACT

Over recent years, there has been a rapid development of deep learning (DL) in both industry and academia fields. However, finding the optimal hyperparameters of a DL model often needs high computational cost and human expertise. To mitigate the above issue, evolutionary computation (EC) as a powerful heuristic search approach has shown significant merits in the automated design of DL models, so-called evolutionary deep learning (EDL). This paper aims to analyze EDL from the perspective of automated machine learning (AutoML). Specifically, we firstly illuminate EDL from machine learning and EC and regard EDL as an optimization problem. According to the DL pipeline, we systematically introduce EDL methods ranging from feature engineering, model generation, to model deployment with a new taxonomy (i.e., what and how to evolve/optimize), and focus on the discussions of solution representation and search paradigm in handling the optimization problem by EC. Finally, key applications, open issues and potentially promising lines of future research are suggested. This survey has reviewed recent developments of EDL and offers insightful guidelines for the development of EDL.

Motivation & Objective

• To provide a systematic review of evolutionary deep learning (EDL) as a solution to the high computational and expert-dependent cost of hyperparameter and architecture tuning in deep learning.
• To establish a new taxonomy for EDL based on 'what', 'where', and 'how' to evolve or optimize within the deep learning pipeline.
• To analyze solution representation and search paradigms in EC-based EDL across feature engineering, model generation, and model deployment stages.
• To identify key challenges in EDL, including fair benchmarking, interpretability, and scalability, and to propose future research directions.

Proposed method

• The paper formulates EDL as a complex optimization problem, leveraging evolutionary computation (EC) to automate the search for optimal deep learning configurations.
• It introduces a novel taxonomy that categorizes EDL methods by the component being evolved (e.g., features, architectures, hyperparameters) and the stage of the DL pipeline.
• The survey analyzes various solution representations (e.g., genotype-to-phenotype mapping) and search paradigms (e.g., multi-objective, steady-state, steady-state with local search) used in EC for EDL.
• It compares EC-based approaches with non-EC methods (e.g., reinforcement learning, gradient-based) in terms of robustness, search capability, and performance across benchmarks.
• The paper evaluates EDL applications in real-world domains such as image recognition, NLP, and medical diagnostics, highlighting performance gains and limitations.
• It identifies and discusses open issues such as lack of standardized benchmarks, poor interpretability, and challenges in handling noisy or small-scale datasets.

Figure 1 . An overview of DL, driven by domain knowledge or evolutionary computation, where the life of DL gets through problem, data collection, feature engineering, model generation and model deployment.

Experimental results

Research questions

• RQ1 How can evolutionary computation be systematically applied to automate deep learning pipelines across feature engineering, model generation, and deployment?
• RQ2 What are the most effective solution representations and search paradigms for EC-based EDL across different stages of the DL pipeline?
• RQ3 How do EC-based EDL methods compare to non-EC alternatives (e.g., reinforcement learning, gradient-based) in terms of performance, robustness, and efficiency?
• RQ4 What are the key challenges in fair benchmarking, interpretability, and scalability of EDL methods in real-world applications?
• RQ5 What future research directions are most promising for advancing end-to-end, scalable, and interpretable evolutionary deep learning?

Key findings

• EDL methods demonstrate competitive performance and robustness in hyperparameter and architecture search, often outperforming manual design and some non-EC approaches on standard benchmarks like CIFAR-10 and ImageNet.
• The lack of uniform benchmarks and evaluation criteria leads to unfair comparisons across EDL methods, particularly in NAS and model compression, where different regularization techniques (e.g., cutout, ScheduledDropPath) significantly affect performance.
• Interpretability remains a major challenge, as EC-based EDL is largely a black-box process with limited theoretical justification for why certain features or architectures are selected.
• While EDL excels in image-related tasks, it lags behind human-designed models (e.g., GPT-2, Transformer-XL) in natural language processing, indicating a performance gap in NLP applications.
• Current EDL frameworks struggle with real-world data challenges such as label noise, data imbalance, and small-scale datasets, suggesting the need to integrate self-supervised or unsupervised learning techniques.
• End-to-end EDL remains a significant challenge due to the complexity of jointly optimizing all pipeline components, but it is identified as a promising future direction for full automation in AutoML.

Figure 2 . Total publications and citations referring to EC & DL by years from Web of Science until July 2022.