[Paper Review] Reinforcement Learning with Deep Energy-Based Policies

Tuomas Haarnoja, Haoran Tang+2 more|arXiv (Cornell University)|Feb 27, 2017

Reinforcement Learning in Robotics40 references434 citations

TL;DR

Introduces soft Q-learning with energy-based policies to represent multimodal stochastic actions in continuous domains, using amortized Stein variational gradient descent to sample from the policy and linking to actor-critic methods.

ABSTRACT

We propose a method for learning expressive energy-based policies for continuous states and actions, which has been feasible only in tabular domains before. We apply our method to learning maximum entropy policies, resulting into a new algorithm, called soft Q-learning, that expresses the optimal policy via a Boltzmann distribution. We use the recently proposed amortized Stein variational gradient descent to learn a stochastic sampling network that approximates samples from this distribution. The benefits of the proposed algorithm include improved exploration and compositionality that allows transferring skills between tasks, which we confirm in simulated experiments with swimming and walking robots. We also draw a connection to actor-critic methods, which can be viewed performing approximate inference on the corresponding energy-based model.

Motivation & Objective

• Motivate learning expressive stochastic policies in continuous state and action spaces.
• Extend maximum entropy policy search to arbitrary policy distributions via energy-based models.
• Develop a practical, scalable algorithm (soft Q-learning) with approximate sampling for deep networks.
• Show benefits in exploration and compositional transfer of skills across tasks.

Proposed method

• Formulate policies as energy-based models with pi(a|s) ∝ exp(-E(s,a)).
• Use soft Q-functions to connect energy-based policies to entropy-regularized RL (Theorem 1).
• Derive soft Bellman equation and soft Q-iteration for fixed-point convergence (Theorem 3).
• Implement practical soft Q-learning with deep networks and a stochastic sampling network via amortized SVGD (fφ) to sample from the policy.
• Optimize Qθ with stochastic gradient methods using replay buffers and target networks.
• Treat the sampling network as an actor, establishing connections to actor-critic frameworks.

Experimental results

Research questions

• RQ1 Can energy-based policies represent complex, multimodal action distributions in continuous domains?
• RQ2 Does maximizing entropy over trajectories improve exploration and enable better pretraining for skill transfer?
• RQ3 How can we efficiently sample from energy-based policies in high dimensions and integrate this with deep RL?
• RQ4 What are the connections between entropy-regularized RL, Q-learning, and actor-critic methods?

Key findings

• The method learns multimodal stochastic policies that align with the energy landscape, enabling diverse behaviors (e.g., multiple goals).
• Soft Q-learning improves exploration in multimodal reward landscapes compared to deterministic methods like DDPG.
• Maximum entropy policies serve as better initializations for finetuning to new skills than standard pretraining.
• Amortized SVGD provides fast, unbiased samples from energy-based policies and reveals an actor-critic interpretation of the approach.
• Empirical results in simulated swimming and quadrupedal walking tasks demonstrate improved exploration and transfer potential.