[Paper Review] Learning adaptive differential evolution algorithm from optimization experiences by policy gradient

Jianyong Sun, Xin Liu+2 more|arXiv (Cornell University)|Feb 6, 2021

Metaheuristic Optimization Algorithms Research55 references4 citations

TL;DR

This paper proposes a reinforcement learning-based adaptive differential evolution (LDE) algorithm that learns optimal control parameters (F, CR, N) from optimization experiences using policy gradient with an LSTM controller. The method models parameter control as a Markov decision process and achieves competitive performance on CEC’13 and CEC’17 test suites, demonstrating the effectiveness of experience-based, adaptive parameter tuning in evolutionary computation.

ABSTRACT

Differential evolution is one of the most prestigious population-based stochastic optimization algorithm for black-box problems. The performance of a differential evolution algorithm depends highly on its mutation and crossover strategy and associated control parameters. However, the determination process for the most suitable parameter setting is troublesome and time-consuming. Adaptive control parameter methods that can adapt to problem landscape and optimization environment are more preferable than fixed parameter settings. This paper proposes a novel adaptive parameter control approach based on learning from the optimization experiences over a set of problems. In the approach, the parameter control is modeled as a finite-horizon Markov decision process. A reinforcement learning algorithm, named policy gradient, is applied to learn an agent (i.e. parameter controller) that can provide the control parameters of a proposed differential evolution adaptively during the search procedure. The differential evolution algorithm based on the learned agent is compared against nine well-known evolutionary algorithms on the CEC'13 and CEC'17 test suites. Experimental results show that the proposed algorithm performs competitively against these compared algorithms on the test suites.

Motivation & Objective

• Address the challenge of selecting optimal control parameters (F, CR, N) in differential evolution, which are typically problem-specific and hard to tune manually.
• Overcome limitations of fixed or heuristic parameter control by enabling dynamic, adaptive parameter adjustment during search based on real-time feedback.
• Develop a parameter controller that learns from historical optimization experiences across multiple problems, rather than relying on online feedback from a single run.
• Improve the robustness, efficiency, and performance of differential evolution by leveraging reinforcement learning to discover stable, generalizable parameter policies.
• Enable the algorithm to adaptively balance exploration and exploitation by learning when to use higher or lower values of F and CR based on search progress and problem characteristics.

Proposed method

• Model the adaptive parameter control problem as a finite-horizon Markov Decision Process (MDP), where states represent optimization progress and actions are parameter settings.
• Use a Long Short-Term Memory (LSTM) recurrent neural network as the policy network to encode historical optimization states and generate control parameters (F, CR, N).
• Train the LSTM policy using the policy gradient reinforcement learning algorithm to maximize cumulative reward based on convergence quality and diversity metrics.
• Collect optimization experiences from training on the CEC’13 test suite to pre-train the controller before deployment on unseen problems.
• Integrate the trained controller into a standard DE framework, allowing dynamic parameter updates during each generation of the search process.
• Define state features (e.g., fitness range, population diversity, convergence rate) as inputs to the LSTM to guide adaptive parameter selection.

Experimental results

Research questions

• RQ1 Can a reinforcement learning agent learn to adaptively control DE parameters (F, CR, N) based on optimization experiences across multiple problems?
• RQ2 Does a policy gradient-trained LSTM controller outperform traditional fixed or heuristic parameter control strategies in terms of convergence speed and solution quality?
• RQ3 How generalizable is the learned parameter policy across different problem types, especially when tested on unseen CEC’17 benchmark functions?
• RQ4 What is the impact of training on a specific set of problems (e.g., CEC’13) on the controller’s performance when applied to different or higher-dimensional problems?
• RQ5 Can experience-based learning lead to more stable and robust parameter adaptation compared to online feedback-only methods?

Key findings

• The proposed LDE algorithm achieved competitive performance against nine well-known evolutionary algorithms on both the CEC’13 and CEC’17 test suites, demonstrating strong generalization and robustness.
• The policy gradient-trained LSTM controller successfully learned to balance exploration and exploitation by dynamically adjusting F and CR based on problem landscape and search progress.
• Training the controller on 30D problems required substantial computational resources, with significant CPU/GPU time, indicating high training cost and scalability challenges.
• The learned controller showed limited performance on problems with features unrelated to the training set, suggesting generalization is constrained by training data similarity.
• The time complexity of the LDE algorithm is higher than baseline DE variants due to the inference overhead of the LSTM policy network.
• Despite its computational cost, the LDE method outperformed several state-of-the-art algorithms, indicating the value of experience-based, adaptive parameter control in evolutionary computation.