[Paper Review] Maximum Entropy RL (Provably) Solves Some Robust RL Problems

Benjamin Eysenbach, Sergey Levine|arXiv (Cornell University)|Mar 10, 2021

Reinforcement Learning in RoboticsComputer Science57 references28 citations

TL;DR

MaxEnt RL provides a provable lower bound on a robust RL objective, yielding policies that are robust to certain disturbances in dynamics and rewards without extra robustness machinery.

ABSTRACT

Many potential applications of reinforcement learning (RL) require guarantees that the agent will perform well in the face of disturbances to the dynamics or reward function. In this paper, we prove theoretically that maximum entropy (MaxEnt) RL maximizes a lower bound on a robust RL objective, and thus can be used to learn policies that are robust to some disturbances in the dynamics and the reward function. While this capability of MaxEnt RL has been observed empirically in prior work, to the best of our knowledge our work provides the first rigorous proof and theoretical characterization of the MaxEnt RL robust set. While a number of prior robust RL algorithms have been designed to handle similar disturbances to the reward function or dynamics, these methods typically require additional moving parts and hyperparameters on top of a base RL algorithm. In contrast, our results suggest that MaxEnt RL by itself is robust to certain disturbances, without requiring any additional modifications. While this does not imply that MaxEnt RL is the best available robust RL method, MaxEnt RL is a simple robust RL method with appealing formal guarantees.

Motivation & Objective

• Motivate robust RL needs where disturbances to dynamics or rewards can occur in real environments.
• Theoretically characterize how MaxEnt RL can yield robust policies under such disturbances.
• Show how maximizing MaxEnt RL relates to a pessimistic robust objective and quantify robustness sets.

Proposed method

• Define the MaxEnt RL objective J_MaxEnt with an entropy term and a balancing coefficient alpha.
• Prove robustness results: (i) robustness to reward perturbations (Theorem 4.1) and (ii) robustness to dynamics perturbations using a pessimistic reward bar{r} (Equation 3) and a divergence-based robustness set (Equation 5).
• Characterize robust sets tilde{R}(pi) and tilde{P}(pi) and relate epsilon to policy entropy (Lemma 4.3).
• Provide corollaries linking MaxEnt RL to lower bounds on unregularized robust objectives (Corollary 4.2.1).
• Offer worked examples to build intuition about reward and dynamics robustness.
• Conduct numerical simulations comparing MaxEnt RL to prior robust methods and standard RL.

Experimental results

Research questions

• RQ1 Can MaxEnt RL maximize a lower bound on a robust RL objective under reward and dynamics perturbations?
• RQ2 What are the robust sets for rewards and dynamics for which MaxEnt RL guarantees hold?
• RQ3 How does the entropy coefficient affect robustness and the size of the robust set?
• RQ4 Do empirical results support the theoretical robustness claims in practical tasks?

Key findings

• MaxEnt RL provably maximizes a lower bound on a robust RL objective when applied to a pessimistic reward function.
• The robustness budget epsilon is lower-bounded by the policy entropy, linking entropy to robustness level.
• MaxEnt RL policies learn multiple routes, offering resilience to disturbances in dynamics or rewards and performing competitively with specialized robust methods.
• The analysis and experiments demonstrate robustness is stronger with larger entropy coefficients, and robustness extends to adversarial perturbations of dynamics.
• Empirical results show MaxEnt RL can outperform or match prior robust RL methods on benchmark tasks while remaining conceptually simpler.