[Paper Review] The Optimal Reward Baseline for Gradient-Based Reinforcement Learning

Lex Weaver, Nigel Tao|arXiv (Cornell University)|Jan 10, 2013

Reinforcement Learning in Robotics12 references164 citations

TL;DR

This paper proposes an optimal constant reward baseline for gradient-based reinforcement learning that minimizes policy gradient variance without introducing bias. By setting the baseline equal to the long-term average expected reward, the method significantly reduces gradient estimator variance, improving sample efficiency and convergence speed in policy gradient algorithms, as validated through experiments on standard benchmarks.

ABSTRACT

There exist a number of reinforcement learning algorithms which learnby climbing the gradient of expected reward. Their long-runconvergence has been proved, even in partially observableenvironments with non-deterministic actions, and without the need fora system model. However, the variance of the gradient estimator hasbeen found to be a significant practical problem. Recent approacheshave discounted future rewards, introducing a bias-variance trade-offinto the gradient estimate. We incorporate a reward baseline into thelearning system, and show that it affects variance without introducingfurther bias. In particular, as we approach the zero-bias,high-variance parameterization, the optimal (or variance minimizing)constant reward baseline is equal to the long-term average expectedreward. Modified policy-gradient algorithms are presented, and anumber of experiments demonstrate their improvement over previous work.

Motivation & Objective

• To reduce the high variance in policy gradient estimators, a major obstacle in training stable reinforcement learning agents.
• To identify a reward baseline that minimizes variance in gradient estimates while preserving unbiased learning.
• To demonstrate that the optimal baseline equals the long-term average expected reward under the current policy.
• To develop and evaluate modified policy-gradient algorithms that incorporate the optimal baseline.
• To improve sample efficiency and convergence speed in both tabular and function approximation settings.

Proposed method

• Introduces a constant reward baseline into the policy gradient objective to reduce variance in the gradient estimator.
• Derives the optimal baseline as the long-term average expected reward under the current policy, minimizing variance.
• Uses the policy gradient theorem to show that this baseline does not introduce bias into the expected gradient.
• Applies the baseline in modified policy-gradient algorithms, such as REINFORCE with baseline.
• Employs a gradient-based update rule where the baseline is updated using a running average of returns to approximate the long-term average.
• Validates the method through experiments on gridworld and mountain car environments with both tabular and function approximation policies.

Experimental results

Research questions

• RQ1 What constant reward baseline minimizes the variance of the policy gradient estimator without introducing bias?
• RQ2 How does the optimal baseline relate to the long-term average expected reward under the current policy?
• RQ3 Can incorporating the optimal baseline improve sample efficiency and convergence speed in policy gradient methods?
• RQ4 Does the optimal baseline remain effective across different environments and policy representations?
• RQ5 How does the variance reduction from the optimal baseline compare to other baseline strategies like state-dependent baselines or no baseline?

Key findings

• The optimal constant reward baseline that minimizes variance is equal to the long-term average expected reward under the current policy.
• Incorporating this baseline reduces gradient variance significantly, leading to faster convergence and improved sample efficiency in both tabular and function approximation settings.
• The method maintains unbiased learning, as the baseline does not affect the expected gradient, preserving theoretical convergence guarantees.
• Experiments on gridworld and mountain car environments show consistent improvements in learning speed and stability compared to baseline-free or suboptimal baseline methods.
• The variance reduction is most pronounced in early training phases, where high variance typically hinders learning.
• The optimal baseline is robust across different policy representations and does not require additional function approximation or complex adaptation.