[Paper Review] Understanding the impact of entropy on policy optimization

Zafarali Ahmed, Nicolas Le Roux+2 more|arXiv (Cornell University)|Nov 27, 2018

Reinforcement Learning in Robotics39 references44 citations

TL;DR

The paper analyzes how entropy regularization shapes the optimization landscape in policy optimization, showing entropy can smooth the objective and enable larger learning rates, with effects that are environment-specific.

ABSTRACT

Entropy regularization is commonly used to improve policy optimization in reinforcement learning. It is believed to help with \emph{exploration} by encouraging the selection of more stochastic policies. In this work, we analyze this claim using new visualizations of the optimization landscape based on randomly perturbing the loss function. We first show that even with access to the exact gradient, policy optimization is difficult due to the geometry of the objective function. Then, we qualitatively show that in some environments, a policy with higher entropy can make the optimization landscape smoother, thereby connecting local optima and enabling the use of larger learning rates. This paper presents new tools for understanding the optimization landscape, shows that policy entropy serves as a regularizer, and highlights the challenge of designing general-purpose policy optimization algorithms.

Motivation & Objective

• Investigate whether entropy regularization affects policy optimization beyond gradient-noise reduction by altering the objective geometry.
• Develop visualization tools to analyze local geometry (gradient and curvature) of the RL objective.
• Assess whether higher policy entropy leads to a smoother landscape that connects local optima and facilitates learning.

Proposed method

• Introduce random perturbation-based visualization of objective geometry to classify local regions (local optimum, saddle, flat).
• Combine linear interpolations and random-direction probes to infer gradient and Hessian information from local samples.
• Apply entropy-augmented rewards to policy gradient objectives and analyze changes in the optimization landscape.
• Test in discrete gridworld environments with exact gradient to isolate landscape effects from gradient variance.
• Extend analysis to continuous control with Gaussian policies to study entropy impact on learning dynamics and curvature.
• Compare true objective vs. entropy-augmented objective to understand how stochastic policies influence optimization paths.

Experimental results

Research questions

• RQ1 Does entropy regularization modify the geometry of the policy optimization landscape beyond reducing gradient estimate variance?
• RQ2 Can higher entropy policies smooth the objective and connect local optima, enabling larger learning rates?
• RQ3 How does entropy affect learning speed and final policy quality across different environments?
• RQ4 Is the impact of entropy on the objective landscape environment-dependent, and if so, why?
• RQ5 What mechanisms (e.g., curvature damping) explain observed improvements with higher entropy policies?

Key findings

• Policy optimization difficulty is strongly linked to the geometry of the objective, not just gradient estimation noise.
• Entropy regularization tends to smooth the objective, connect local optima, and allow larger learning rates in some environments.
• In deterministic gridworlds, higher entropy reveals directions of improvement and reduces flat regions, aiding optimization.
• In continuous control tasks, higher entropy can speed up learning and improve final performance in Hopper and Walker, but effects are environment-specific (HalfCheetah shows less pronounced benefits).
• Curvature fluctuations during training decrease with higher entropy in some environments, supporting faster, more stable optimization with larger learning rates.
• Final policy landscapes under high entropy show fewer negative-curvature directions, indicating possible movement toward flatter regions.