[Paper Review] Relative Entropy Regularized Policy Iteration

Abbas Abdolmaleki, Jost Tobias Springenberg+6 more|arXiv (Cornell University)|Dec 5, 2018

Reinforcement Learning in Robotics35 references45 citations

TL;DR

An off-policy actor-critic RL method that alternates Q-value estimation, local non-parametric policy improvement with KL regularization, and parametric policy fitting with decoupled Gaussian updates, achieving strong results across multiple continuous-control benchmarks.

ABSTRACT

We present an off-policy actor-critic algorithm for Reinforcement Learning (RL) that combines ideas from gradient-free optimization via stochastic search with learned action-value function. The result is a simple procedure consisting of three steps: i) policy evaluation by estimating a parametric action-value function; ii) policy improvement via the estimation of a local non-parametric policy; and iii) generalization by fitting a parametric policy. Each step can be implemented in different ways, giving rise to several algorithm variants. Our algorithm draws on connections to existing literature on black-box optimization and 'RL as an inference' and it can be seen either as an extension of the Maximum a Posteriori Policy Optimisation algorithm (MPO) [Abdolmaleki et al., 2018a], or as an extension of Trust Region Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES) [Abdolmaleki et al., 2017b; Hansen et al., 1997] to a policy iteration scheme. Our comparison on 31 continuous control tasks from parkour suite [Heess et al., 2017], DeepMind control suite [Tassa et al., 2018] and OpenAI Gym [Brockman et al., 2016] with diverse properties, limited amount of compute and a single set of hyperparameters, demonstrate the effectiveness of our method and the state of art results. Videos, summarizing results, can be found at goo.gl/HtvJKR .

Motivation & Objective

• Develop a data-efficient off-policy actor-critic framework for continuous control.
• Combine Q-function estimation with a local non-parametric policy improvement step.
• Introduce a parametric policy fitting step with KL-based regularization to ensure stable learning.
• Enable decoupled mean and covariance updates for Gaussian policies to prevent premature convergence.
• Demonstrate robustness across diverse benchmarks using a single set of hyperparameters.

Proposed method

• Policy evaluation by learning a parametric Q-function with TD learning and a target network.
• Policy improvement by constructing a local non-parametric action distribution via reweighted samples using Q-values.
• Projection back to a parametric policy through weighted maximum likelihood with KL regularization (softmax-based weights).
• Optionally transform weights via exponential or ranking-based schemes; solve for a temperature parameter via a convex dual.
• Fitting an improved Gaussian policy with decoupled updates for mean and covariance to prevent premature convergence.
• Regularization via KL constraints on the mean and covariance to control policy updates; coordinate ascent optimization.

Experimental results

Research questions

• RQ1 How does KL-regularized policy improvement affect stability and performance in off-policy actor-critic settings?
• RQ2 Can a decoupled mean/covariance update for Gaussian policies improve learning stability and prevent premature convergence?
• RQ3 How does this framework perform across diverse continuous control tasks (Control Suite, Parkour, OpenAI Gym) using a single hyperparameter set?
• RQ4 What is the impact of different Q-function estimation strategies (e.g., TD0 vs Retrace) on final performance in complex tasks?
• RQ5 How does this method compare to established baselines such as DDPG, SVG, and SAC on high-dimensional tasks?

Key findings

• The method achieves strong performance on 31 continuous control tasks across multiple benchmarks using a single hyperparameter set.
• Decoupled mean/covariance updates for Gaussian policies help avoid premature convergence and improve stability and performance.
• KL constraints on both mean and covariance are important for reliable learning across tasks; without them, learning can be unstable.
• Retrace-based policy evaluation speeds up learning in challenging Parkour tasks compared to TD0.
• In OpenAI Gym tasks, the method attains higher asymptotic performance than SAC with comparable sample efficiency.