[Paper Review] The Mirage of Action-Dependent Baselines in Reinforcement Learning

George Tucker, Surya Bhupatiraju+4 more|arXiv (Cornell University)|Feb 27, 2018

Reinforcement Learning in RoboticsComputer Science36 references45 citations

TL;DR

The paper decomposes policy gradient variance and shows that learned state-action-dependent baselines do not significantly reduce variance over state-dependent baselines in common benchmarks, uncovering implementation-induced bias and proposing horizon-aware value functions as a practical improvement.

ABSTRACT

Policy gradient methods are a widely used class of model-free reinforcement learning algorithms where a state-dependent baseline is used to reduce gradient estimator variance. Several recent papers extend the baseline to depend on both the state and action and suggest that this significantly reduces variance and improves sample efficiency without introducing bias into the gradient estimates. To better understand this development, we decompose the variance of the policy gradient estimator and numerically show that learned state-action-dependent baselines do not in fact reduce variance over a state-dependent baseline in commonly tested benchmark domains. We confirm this unexpected result by reviewing the open-source code accompanying these prior papers, and show that subtle implementation decisions cause deviations from the methods presented in the papers and explain the source of the previously observed empirical gains. Furthermore, the variance decomposition highlights areas for improvement, which we demonstrate by illustrating a simple change to the typical value function parameterization that can significantly improve performance.

Motivation & Objective

• Motivate a precise understanding of variance reduction from state-action-dependent baselines in policy gradient methods.
• Decompose the policy gradient variance to identify where variance reduction can realistically occur.
• Evaluate variance components on synthetic and benchmark tasks to assess practical benefits of state-action-dependent baselines.

Proposed method

• Provide a variance decomposition of the policy gradient estimator with a state-action-dependent baseline (Eq. 2 and Eq. 3).
• Analyze variance terms Sigma_tau, Sigma_a, and Sigma_s and identify conditions under which Sigma_a is impactful.
• Empirically measure variance components on LQG and continuous control tasks with oracle and learned baselines.
• Review open-source implementations to identify bias-inducing implementation details.
• Propose horizon-aware value function parameterization to better match finite-horizon tasks.

Experimental results

Research questions

• RQ1 Do learned state-action-dependent baselines reduce the policy gradient variance more than state-dependent baselines in benchmark tasks?
• RQ2 What is the relative magnitudes of the variance components (Sigma_tau, Sigma_a, Sigma_s) across tasks and estimators?
• RQ3 How do implementation details and value function approximations affect the observed benefits of action-dependent baselines?
• RQ4 Can horizon-aware value functions provide practical improvements without biasing the gradient estimates?

Key findings

• A learned state-action-dependent baseline does not significantly reduce variance over a learned state-dependent baseline on tested tasks.
• The variance reduced by state-action-dependent baselines is often dominated by variance from the value function approximator and the baseline itself.
• Some reported gains from state-action-dependent baselines arise from implementation choices that introduce bias, not unbiased variance reduction.
• Function approximation gaps for V(s) and phi(s,a) contribute more to variance than action dependence of the baseline in typical benchmarks.
• A horizon-aware value function parameterization yields performance improvements over baselines in experiments.
• Improving value function approximation is a more promising path for variance reduction than adopting action-dependent baselines under current methods.