[Paper Review] Beyond variance reduction: Understanding the true impact of baselines on policy optimization

Wesley Chung, Valentin Thomas+2 more|arXiv (Cornell University)|Aug 31, 2020

Advanced Bandit Algorithms Research35 references4 citations

TL;DR

This paper demonstrates that baselines in policy optimization affect convergence beyond variance reduction, with negative baselines inducing premature policy commitment and positive baselines preserving exploration. It shows that even with identical gradient variance, baseline choice can cause natural policy gradient to converge to suboptimal policies, challenging standard optimization assumptions in reinforcement learning.

ABSTRACT

Bandit and reinforcement learning (RL) problems can often be framed as optimization problems where the goal is to maximize average performance while having access only to stochastic estimates of the true gradient. Traditionally, stochastic optimization theory predicts that learning dynamics are governed by the curvature of the loss function and the noise of the gradient estimates. In this paper we demonstrate that this is not the case for bandit and RL problems. To allow our analysis to be interpreted in light of multi-step MDPs, we focus on techniques derived from stochastic optimization principles (e.g., natural policy gradient and EXP3) and we show that some standard assumptions from optimization theory are violated in these problems. We present theoretical results showing that, at least for bandit problems, curvature and noise are not sufficient to explain the learning dynamics and that seemingly innocuous choices like the baseline can determine whether an algorithm converges. These theoretical findings match our empirical evaluation, which we extend to multi-state MDPs.

Motivation & Objective

• To investigate the impact of baselines on policy optimization dynamics beyond variance reduction.
• To challenge the assumption that curvature and gradient noise fully determine learning dynamics in bandit and MDP settings.
• To identify conditions under which natural policy gradient fails to converge due to baseline choice.
• To analyze the role of on-policy sampling in inducing convergence issues via a feedback loop between poor updates and degraded policies.
• To propose off-policy sampling with sufficient action coverage as a remedy to guarantee convergence in probability.

Proposed method

• Theoretical analysis of stochastic gradient methods in multi-arm bandit and MDP settings, focusing on natural policy gradient and EXP3.
• Derivation of gradient variance and bias under different baseline choices, showing identical variance for baselines symmetrically offset from the minimum-variance baseline.
• Construction of a three-armed bandit counterexample where the minimum-variance baseline causes convergence to a suboptimal deterministic policy.
• Introduction of a baseline-dependent feedback loop in on-policy sampling, where poor updates degrade policy quality, worsening future updates.
• Proposed solution: off-policy sampling with uniform action probability to decouple sampling from policy updates and ensure convergence in probability.
• Theoretical proof that sufficient action coverage in off-policy sampling guarantees convergence to the optimal policy, regardless of baseline choice.

Experimental results

Research questions

• RQ1 Can baseline choice affect policy optimization convergence independently of gradient variance?
• RQ2 Why do some baselines lead to premature policy commitment while others preserve exploration, even with identical gradient variance?
• RQ3 Can natural policy gradient fail to converge to the optimal policy due to baseline choice, despite low variance?
• RQ4 What role does on-policy sampling play in creating a feedback loop that hinders convergence?
• RQ5 Is off-policy sampling with sufficient action coverage sufficient to guarantee convergence in probability?

Key findings

• Two baselines with identical gradient variance can lead to qualitatively different learning dynamics: one induces rapid policy commitment to a deterministic, suboptimal policy, while the other preserves exploration and entropy.
• A three-armed bandit example demonstrates that the minimum-variance baseline can cause natural policy gradient to converge to a suboptimal policy for any positive stepsize.
• The failure of natural policy gradient under standard assumptions indicates that curvature and noise alone cannot explain learning dynamics in bandit and MDP problems.
• On-policy sampling creates a vicious cycle where poor updates lead to worse policies, which in turn produce worse updates, undermining convergence.
• Off-policy sampling with sufficient action coverage—ensuring all actions are sampled with bounded probability—guarantees convergence in probability to the optimal policy.
• Theoretical analysis confirms that baselines do not affect the bias of the natural policy gradient estimator, but their choice still alters convergence behavior due to non-convexity and feedback dynamics.