[Paper Review] Sample-Efficient Automated Deep Reinforcement Learning

Jörg K. H. Franke, Gregor Köhler+2 more|arXiv (Cornell University)|Sep 3, 2020

Reinforcement Learning in Robotics39 references4 citations

TL;DR

This paper proposes Sample-Efficient Automated Reinforcement Learning (SEARL), a population-based AutoRL framework that jointly optimizes hyperparameters and neural architecture in off-policy deep RL using shared experience replay. By enabling dynamic hyperparameter adaptation and architecture evolution while training multiple agents in parallel, SEARL reduces environment interactions by up to 10× compared to random search and PBT, achieving superior or comparable performance with significantly improved sample efficiency.

ABSTRACT

Despite significant progress in challenging problems across various domains, applying state-of-the-art deep reinforcement learning (RL) algorithms remains challenging due to their sensitivity to the choice of hyperparameters. This sensitivity can partly be attributed to the non-stationarity of the RL problem, potentially requiring different hyperparameter settings at various stages of the learning process. Additionally, in the RL setting, hyperparameter optimization (HPO) requires a large number of environment interactions, hindering the transfer of the successes in RL to real-world applications. In this work, we tackle the issues of sample-efficient and dynamic HPO in RL. We propose a population-based automated RL (AutoRL) framework to meta-optimize arbitrary off-policy RL algorithms. In this framework, we optimize the hyperparameters and also the neural architecture while simultaneously training the agent. By sharing the collected experience across the population, we substantially increase the sample efficiency of the meta-optimization. We demonstrate the capabilities of our sample-efficient AutoRL approach in a case study with the popular TD3 algorithm in the MuJoCo benchmark suite, where we reduce the number of environment interactions needed for meta-optimization by up to an order of magnitude compared to population-based training.

Motivation & Objective

• To address the high sample complexity of hyperparameter optimization (HPO) in deep reinforcement learning (RL), which limits real-world deployment.
• To overcome the non-stationarity of RL by enabling dynamic hyperparameter adaptation across training phases.
• To integrate neural architecture search (NAS) into gradient-based deep RL with minimal computational overhead.
• To develop a fair evaluation protocol for AutoRL that accounts for actual environment interaction costs.
• To demonstrate that joint optimization of hyperparameters and architecture via shared experience can achieve superior sample efficiency.

Proposed method

• A population-based evolutionary framework that simultaneously trains multiple RL agents with diverse hyperparameters and architectures.
• Shared replay memory across the population to increase sample efficiency by reusing collected experience.
• Evolvable neural networks that preserve trained weights while adapting depth, width, and activation functions during training.
• Dynamic hyperparameter tuning via evolutionary mutation, enabling configuration schedules rather than static settings.
• Joint optimization of off-policy RL algorithms (e.g., TD3, DQN) with both algorithmic and architectural hyperparameters.
• Synchronous evolutionary loop that ensures coordination between population members while minimizing redundant environment interactions.

Experimental results

Research questions

• RQ1 Can shared experience replay across a population of RL agents significantly reduce the number of environment interactions required for meta-optimization?
• RQ2 Does dynamic hyperparameter adaptation during training improve final performance compared to static configurations?
• RQ3 Can neural architecture evolution be effectively integrated into gradient-based deep RL with minimal computational cost?
• RQ4 How does SEARL compare in sample efficiency and final performance to random search and population-based training (PBT)?
• RQ5 Can SEARL generalize across different RL algorithms and environments, including on-policy and off-policy setups?

Key findings

• SEARL reduces the number of required environment interactions for meta-optimization by up to an order of magnitude (10×) compared to random search and PBT in the MuJoCo benchmark.
• SEARL achieves performance on par with or better than the best random search and PBT configurations in 4 out of 5 MuJoCo environments.
• The use of shared replay memory enables SEARL to perform AutoRL at nearly the same sample cost as training a single agent, making it almost 'free' in terms of environment interaction overhead.
• SEARL discovers effective hyperparameter schedules that adapt across training phases, addressing the non-stationarity of RL problems.
• SEARL generalizes well to other algorithms, outperforming the best random search configuration in all but one of five Atari environments when tuning Rainbow DQN.
• The framework requires only twice the environment interactions of a single TD3 run while simultaneously optimizing hyperparameters and architecture, demonstrating strong sample efficiency.