[Paper Review] Maximum a Posteriori Policy Optimisation

Abbas Abdolmaleki, Jost Tobias Springenberg+4 more|arXiv (Cornell University)|Jun 14, 2018

Reinforcement Learning in Robotics28 references167 citations

TL;DR

MPO introduces an off-policy reinforcement learning algorithm based on coordinate ascent on a relative-entropy objective, yielding data-efficient and robust policy optimization through an EM-like E-step (reweighting samples) and M-step (supervised MAP update).

ABSTRACT

We introduce a new algorithm for reinforcement learning called Maximum aposteriori Policy Optimisation (MPO) based on coordinate ascent on a relative entropy objective. We show that several existing methods can directly be related to our derivation. We develop two off-policy algorithms and demonstrate that they are competitive with the state-of-the-art in deep reinforcement learning. In particular, for continuous control, our method outperforms existing methods with respect to sample efficiency, premature convergence and robustness to hyperparameter settings while achieving similar or better final performance.

Motivation & Objective

• Motivate and address sample inefficiency and instability in deep RL for continuous control.
• Develop an off-policy algorithm that combines data efficiency with the stability of on-policy methods.
• Leverage an EM-like framework to separate trajectory reweighting (E-step) from policy updates (M-step).
• Relate MPO to existing RL approaches and demonstrate robustness to hyperparameters.

Proposed method

• Formulate RL as variational inference with a lower bound (ELBO) on policy optimality.
• Use an EM-like coordinate ascent alternating E-step (optimize q(a|s) given a fixed policy) and M-step (update policy parameters via supervised MAP with q-weighting).
• Implement the E-step with either a parametric or non-parametric variational distribution q(a|s); derive a closed-form for q when non-parametric: q(a|s) ∝ π(a|s,θ) exp(Qθ(s,a)/η).
• Introduce hard KL constraints in the E-step to stabilize optimization, or equivalently use a temperature-based regularization parameter α.
• In the M-step, perform a weighted maximum a-posteriori update of the policy parameters with a KL constraint relative to the previous policy to improve generalization.
• Employ Retrace for stable off-policy Q-function evaluation and bootstrapped targets for the Q-network.

Experimental results

Research questions

• RQ1 Does MPO achieve high data efficiency in continuous control while maintaining robustness to hyperparameters?
• RQ2 Can an off-policy EM-style optimization framework match or surpass state-of-the-art methods (TRPO/PPO, DDPG) in sample efficiency and stability?
• RQ3 How does MPO perform on high-dimensional tasks (e.g., 56 DoF humanoid) and under off-policy learning conditions?
• RQ4 What is the impact of non-parametric vs parametric variational distributions for q(a|s) on performance and stability?

Key findings

• MPO attains strong data efficiency and robust learning across a broad set of continuous control tasks.
• MPO outperforms state-of-the-art methods in sample efficiency, premature convergence, and hyperparameter robustness on high-dimensional control problems.
• An off-policy EM-style approach provides stability without requiring gradients of the Q-function for policy updates.
• Using a hard KL constraint in the E-step and a KL constraint in the M-step improves stability and generalization.
• Non-parametric q(a|s) optimization yields a closed-form solution that leverages samples and Q-values to reweight actions.
• Experiments show MPO solving all tasks with relatively modest data (often under 1000 trajectories).