[Paper Review] CEM-RL: Combining evolutionary and gradient-based methods for policy search

Aloïs Pourchot, Olivier Sigaud|arXiv (Cornell University)|Oct 2, 2018

Reinforcement Learning in RoboticsComputer Science30 references95 citations

TL;DR

CEM-RL integrates the Cross-Entropy Method with TD3 to jointly leverage evolutionary exploration and gradient-based policy improvement, achieving competitive or superior performance and stability across continuous control benchmarks.

ABSTRACT

Deep neuroevolution and deep reinforcement learning (deep RL) algorithms are two popular approaches to policy search. The former is widely applicable and rather stable, but suffers from low sample efficiency. By contrast, the latter is more sample efficient, but the most sample efficient variants are also rather unstable and highly sensitive to hyper-parameter setting. So far, these families of methods have mostly been compared as competing tools. However, an emerging approach consists in combining them so as to get the best of both worlds. Two previously existing combinations use either an ad hoc evolutionary algorithm or a goal exploration process together with the Deep Deterministic Policy Gradient (DDPG) algorithm, a sample efficient off-policy deep RL algorithm. In this paper, we propose a different combination scheme using the simple cross-entropy method (CEM) and Twin Delayed Deep Deterministic policy gradient (td3), another off-policy deep RL algorithm which improves over ddpg. We evaluate the resulting method, cem-rl, on a set of benchmarks classically used in deep RL. We show that cem-rl benefits from several advantages over its competitors and offers a satisfactory trade-off between performance and sample efficiency.

Motivation & Objective

• Motivate combining evolutionary strategies with deep reinforcement learning for policy search to balance exploration stability and sample efficiency.
• Propose a concrete method (cem-rl) that couples the cross-entropy method with a TD3-based critic-driven gradient update.
• Evaluate cem-rl against baselines (cem, td3, multi-actor td3) and against an existing hybrid (erl) across standard Mujoco benchmarks.
• Analyze the contribution of the evolutionary component and the gradient-based improvements to performance and stability.

Proposed method

• Use a population of actors sampled from a Gaussian around the current mean policy with covariance Sigma.
• Half of the population is evaluated directly; the other half is improved via gradient steps guided by a TD3/critic before re-evaluation.
• Update the population mean and covariance using the top-performing half (cem updates).
• Integrate a replay buffer and train the critic with new experiences; gradient steps are applied to actors derived from the population.
• Possibly emphasize importance mixing for sampling, with an explicit discussion of resource allocation across environment steps and learning updates.

Experimental results

Research questions

• RQ1 Does cem-rl outperform its components (cem and td3) and a multi-actor variant of td3 across standard continuous-control benchmarks?
• RQ2 How does cem-rl compare to erl in terms of final performance, convergence speed, and learning stability?
• RQ3 Does the combination provide improved sample efficiency and/or robustness to hyperparameters in practice?
• RQ4 To what extent does the evolutionary component contribute beyond simply providing a population-based exploration?
• RQ5 What are the limiting factors or environment characteristics where cem-rl may underperform?

Key findings

• cem-td3 generally outperforms cem, td3, and multi-actor td3 across several Mujoco benchmarks, with reduced variance in learning.
• cem-rl methods (cem-ddpg and cem-td3) outperform erl on multiple environments in the tested settings, with cem-td3 often providing the best final performance and faster convergence.
• Ablation shows that replacing the gradient-consistent TD3 guidance with a shared gradient across actors (multi-actor TD3) reduces performance, indicating the benefit of the combined evolutionary-gradient scheme.
• Compared to erl, cem-td3 frequently yields better stability and final performance, particularly in harder environments like walker2d-v2 and ant-v2.