[Paper Review] Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review

Sergey Levine|ArXiv.org|May 2, 2018

Reinforcement Learning in Robotics41 references374 citations

TL;DR

The paper shows how maximum entropy reinforcement learning and control problems can be cast as probabilistic inference in graphical models, deriving exact inference for deterministic dynamics and variational methods for stochastic dynamics, with connections to deep RL and planning.

ABSTRACT

The framework of reinforcement learning or optimal control provides a mathematical formalization of intelligent decision making that is powerful and broadly applicable. While the general form of the reinforcement learning problem enables effective reasoning about uncertainty, the connection between reinforcement learning and inference in probabilistic models is not immediately obvious. However, such a connection has considerable value when it comes to algorithm design: formalizing a problem as probabilistic inference in principle allows us to bring to bear a wide array of approximate inference tools, extend the model in flexible and powerful ways, and reason about compositionality and partial observability. In this article, we will discuss how a generalization of the reinforcement learning or optimal control problem, which is sometimes termed maximum entropy reinforcement learning, is equivalent to exact probabilistic inference in the case of deterministic dynamics, and variational inference in the case of stochastic dynamics. We will present a detailed derivation of this framework, overview prior work that has drawn on this and related ideas to propose new reinforcement learning and control algorithms, and describe perspectives on future research.

Motivation & Objective

• Provide a unified probabilistic graphical model (PGM) formulation for reinforcement learning and control with an entropy term.
• Show how optimal trajectories arise as inference in this PGM, and derive corresponding backward messages and soft value functions.
• Contrast deterministic and stochastic dynamics, highlighting the need for variational inference to avoid unrealistic risk-seeking dynamics.
• Clarify objective: when entropy is included, recover maximum entropy RL and explain implications for reward design and policy learning.

Proposed method

• Introduce a maximum-entropy extension of the RL/control objective via an auxiliary optimality variable O_t with p(O_t=1|s_t,a_t)=exp(r(s_t,a_t)).
• Formulate a PGM where trajectories are weighted by exp(sum_t r(s_t,a_t)) and perform (exact) inference for deterministic dynamics or variational/inference-based methods for stochastic dynamics.
• Derive backward messages beta_t(s_t,a_t) and beta_t(s_t) to recover p(a_t|s_t,O_1:T) and show relation to soft Q/V functions (Q(s,a)=r(s,a)+V(s')).
• Present log-space backups Q and V, linking to soft Bellman backups in the deterministic case and discussing risk-seeking behavior in stochastic dynamics (addressed via variational corrections).
• Discuss alternative model forms (undirected CRFs, temperature parameter alpha) and discounting, connecting to standard and entropy-regularized RL frameworks.

Experimental results

Research questions

• RQ1 How can reinforcement learning and optimal control be reformulated as probabilistic inference in a graphical model?
• RQ2 What is the behavior and interpretation of entropy-regularized objectives in deterministic versus stochastic dynamics?
• RQ3 How can we compute optimal policies via backward messages in the control-as-inference framework?
• RQ4 How does variational inference address the risk-seeking issue that arises with stochastic dynamics in the maximum-entropy formulation?
• RQ5 How do alternative formulations (CRFs, temperature, discounting) relate to standard RL and maximum entropy RL?

Key findings

• Maximum entropy formulations of RL/control correspond to exact inference with deterministic dynamics and variational inference with stochastic dynamics.
• Optimal policies can be recovered via backward messages β_t(s_t,a_t) and β_t(s_t), leading to soft Q/V functions.
• In log space, the soft Bellman backups reveal the scheduling of exploration through entropy and the risk-seeking effect under stochastic dynamics.
• A variational inference approach fixes dynamics (freezing p(s_{t+1}|s_t,a_t)) and yields robust backups that use expected next-state values, mitigating risk-seeking behavior.
• Alternative formulations (undirected CRFs, temperature parameter) allow interpolation between entropy maximization and standard RL objectives; discounting can be incorporated trivially.