[Paper Review] Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning

Felipe Petroski Such, Vashisht Madhavan+4 more|arXiv (Cornell University)|Dec 18, 2017

Reinforcement Learning in RoboticsComputer Science48 references554 citations

TL;DR

The paper shows a simple gradient-free genetic algorithm can train deep neural networks for reinforcement learning at scales rivaling gradient-based methods on Atari and Humanoid tasks, with novel encoding and exploration techniques.

ABSTRACT

Deep artificial neural networks (DNNs) are typically trained via gradient-based learning algorithms, namely backpropagation. Evolution strategies (ES) can rival backprop-based algorithms such as Q-learning and policy gradients on challenging deep reinforcement learning (RL) problems. However, ES can be considered a gradient-based algorithm because it performs stochastic gradient descent via an operation similar to a finite-difference approximation of the gradient. That raises the question of whether non-gradient-based evolutionary algorithms can work at DNN scales. Here we demonstrate they can: we evolve the weights of a DNN with a simple, gradient-free, population-based genetic algorithm (GA) and it performs well on hard deep RL problems, including Atari and humanoid locomotion. The Deep GA successfully evolves networks with over four million free parameters, the largest neural networks ever evolved with a traditional evolutionary algorithm. These results (1) expand our sense of the scale at which GAs can operate, (2) suggest intriguingly that in some cases following the gradient is not the best choice for optimizing performance, and (3) make immediately available the multitude of neuroevolution techniques that improve performance. We demonstrate the latter by showing that combining DNNs with novelty search, which encourages exploration on tasks with deceptive or sparse reward functions, can solve a high-dimensional problem on which reward-maximizing algorithms (e.g.\ DQN, A3C, ES, and the GA) fail. Additionally, the Deep GA is faster than ES, A3C, and DQN (it can train Atari in ${ aise.17ex\hbox{$\scriptstyle\sim$}}$4 hours on one desktop or ${ aise.17ex\hbox{$\scriptstyle\sim$}}$1 hour distributed on 720 cores), and enables a state-of-the-art, up to 10,000-fold compact encoding technique.

Motivation & Objective

• Assess whether a plain genetic algorithm (GA) can train deep neural networks for challenging RL tasks at scale.
• Compare GA performance to DQN, A3C, and ES across Atari and MuJoCo Humanoid locomotion.
• Explore benefits of novelty search and other neuroevolution techniques in deep RL settings.
• Demonstrate efficient, compressed encoding of large networks evolved by GA.
• Investigate speed and scalability advantages of GA in single-machine and distributed setups.

Proposed method

• Use a simple, gradient-free GA with truncation selection and elitism to evolve neural network weights.
• Mutate offspring with additive Gaussian noise; keep the best individual as elitist; evaluate top individuals with multiple episodes to reduce noise.
• Represent large weight vectors via seed-based encoding to enable compact, scalable distributed training.
• Apply novelty search (GA-NS) by replacing fitness with behavioral novelty in a supervised way on a deceptive task.
• Test two experimental setups: Atari from pixels (4M+ parameter networks) and Humanoid Locomotion in MuJoCo; compare against DQN, ES, and A3C.
• Include a distributed CPU-based and GPU-accelerated GA implementation to assess wall-clock speed and scalability.

Experimental results

Research questions

• RQ1 Can a simple GA train deep neural networks at scales used in deep RL benchmarks (e.g., 4M+ parameters) effectively on Atari and Humanoid tasks?
• RQ2 How does GA performance compare to gradient-based methods (DQN, A3C) and ES across these domains?
• RQ3 Does novelty search improve exploration and performance in deep RL when combined with GA on deceptive or high-dimensional tasks?
• RQ4 Can large networks evolved by GA be encoded compactly, enabling efficient distributed training?
• RQ5 What are the wall-clock time and Computational cost advantages of GA versus other deep RL methods?

Key findings

• GA-trained networks perform comparably to DQN, A3C, and ES on multiple Atari games, with superiority on some titles (e.g., Skiing, Frostbite, Venture).
• GAs can evolve networks with over four million parameters, representing the largest neural networks evolved with a traditional evolutionary algorithm at the time.
• GA runs can be significantly faster in wall-clock time than DQN and A3C, including a desktop setup (~4 hours on 4 GPUs/48 CPUs) and distributed runs (~1 hour on 720 CPUs).
• Novelty search (GA-NS) enables solving a high-dimensional image-based maze that reward-only GA and other baselines fail to solve.
• Random search often outperforms some gradient-based methods on certain games, highlighting that dense local search near the origin can yield strong solutions in some domains.
• GA with novelty exploration demonstrates the value of integrating diversity and quality signals in deep neuroevolution, suggesting potential hybrid approaches with deep RL.