[Paper Review] Neural Architecture Evolution in Deep Reinforcement Learning for Continuous Control

Jörg K. H. Franke, Gregor Köhler+2 more|arXiv (Cornell University)|Oct 28, 2019

Reinforcement Learning in Robotics21 references4 citations

TL;DR

This paper proposes Actor-Critic Neuroevolution (ACN), a method that jointly evolves neural network architectures and policies in continuous control tasks using a novel distillation-based mutation operator. By combining off-policy deep reinforcement learning with neuroevolution, ACN discovers high-performing, sample-efficient architectures—often smaller and more effective than handcrafted ones—while incurring only a minor computational overhead over standard training with fixed architectures.

ABSTRACT

Current Deep Reinforcement Learning algorithms still heavily rely on handcrafted neural network architectures. We propose a novel approach to automatically find strong topologies for continuous control tasks while only adding a minor overhead in terms of interactions in the environment. To achieve this, we combine Neuroevolution techniques with off-policy training and propose a novel architecture mutation operator. Experiments on five continuous control benchmarks show that the proposed Actor-Critic Neuroevolution algorithm often outperforms the strong Actor-Critic baseline and is capable of automatically finding topologies in a sample-efficient manner which would otherwise have to be found by expensive architecture search.

Motivation & Objective

• To address the reliance of deep reinforcement learning on handcrafted neural network architectures in continuous control tasks.
• To enable online, sample-efficient architecture search during training, rather than pre-training or post-hoc search.
• To develop a stable genetic operator for evolving network topology without catastrophic forgetting or performance collapse.
• To integrate neuroevolution with off-policy actor-critic training using a shared global replay buffer.
• To demonstrate that evolved architectures can outperform or match fixed architectures with less environment interaction.

Proposed method

• ACN uses a genetic algorithm to evolve a population of actor and critic networks, each with distinct topologies and parameters.
• A novel distillation-based mutation operator grows network topology and then transfers knowledge from parent to offspring using MSE loss on states sampled from a global replay buffer.
• The mutation process first increases capacity by adding layers or nodes, then stabilizes performance via knowledge distillation from the parent network.
• Off-policy training with TD3 is applied to all individuals in the population, sharing experience in a global replay buffer to improve sample efficiency.
• Tournament selection selects parents for the next generation based on fitness, which is the cumulative return from environment interactions.
• The method supports online architecture evolution, where network topology is modified during training without reinitializing the policy or value function.

Experimental results

Research questions

• RQ1 Can neural network architectures be evolved online during deep reinforcement learning training for continuous control tasks?
• RQ2 Does a distillation-based mutation operator stabilize performance during topology changes in actor-critic networks?
• RQ3 Can evolved architectures outperform or match handcrafted architectures with minimal additional environment interaction?
• RQ4 How does the integration of neuroevolution with off-policy RL affect sample efficiency and final performance?
• RQ5 What architectural patterns emerge in different continuous control environments, and do they align with known optimal designs?

Key findings

• ACN outperforms or matches the performance of TD3 with fixed architectures on all five continuous control benchmarks: HalfCheetah, Ant, Walker2d, Humanoid, and Hopper.
• In Humanoid, ACN achieves substantial performance gains, likely due to its exploratory nature in both architecture and parameter space.
• The best architectures found by ACN are often smaller than the default TD3 architecture—e.g., [80, 80, 88] in HalfCheetah vs. [600, 450]—and still achieve superior or comparable performance.
• In Ant, ACN finds a single-layer architecture with 276 units, half the size of the TD3 default, yet performs comparably.
• The method incurs only a small computational overhead, as the shared replay buffer and evolved architecture reduce total environment interactions needed.
• Experiments show that reinitializing the optimizer and target networks during training (as done in ACN) does not harm performance and can even improve it.