[Paper Review] Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Estimators for Reinforcement Learning

Gregory Farquhar, Shimon Whiteson+1 more|arXiv (Cornell University)|Sep 23, 2019

Reinforcement Learning in Robotics17 references4 citations

TL;DR

This paper proposes Loaded DiCE, a novel objective for estimating any-order derivatives in reinforcement learning with controllable bias-variance trade-offs. By combining arbitrary advantage estimators with a discounting mechanism for distant causal dependencies, it enables low-variance, unbiased higher-order gradient estimation via automatic differentiation, significantly improving performance in meta-RL and complex sequential decision-making tasks.

ABSTRACT

Gradient-based methods for optimisation of objectives in stochastic settings with unknown or intractable dynamics require estimators of derivatives. We derive an objective that, under automatic differentiation, produces low-variance unbiased estimators of derivatives at any order. Our objective is compatible with arbitrary advantage estimators, which allows the control of the bias and variance of any-order derivatives when using function approximation. Furthermore, we propose a method to trade off bias and variance of higher order derivatives by discounting the impact of more distant causal dependencies. We demonstrate the correctness and utility of our objective in analytically tractable MDPs and in meta-reinforcement-learning for continuous control.

Motivation & Objective

• To address the challenge of high-variance and biased higher-order gradient estimators in reinforcement learning and sequential decision-making.
• To enable the use of arbitrary advantage estimators—such as value functions—for controlling bias and variance in any-order derivatives.
• To provide a single, differentiable objective compatible with automatic differentiation that supports higher-order gradient estimation in stochastic computation graphs with Markov structure.
• To introduce a causal discounting mechanism that reduces the impact of distant dependencies, enabling further control over bias and variance in higher-order derivatives.
• To empirically validate the method in analytically tractable MDPs and meta-RL environments, demonstrating improved training stability and performance.

Proposed method

• The paper derives a surrogate objective that, when differentiated multiple times, yields correct estimators for any-order derivatives in Markovian stochastic computation graphs.
• The objective integrates arbitrary advantage estimators (e.g., value functions) into the gradient estimation process, allowing known first-order bias-variance trade-offs to extend to higher-order derivatives.
• A novel causal discounting mechanism is introduced via a hyperparameter λ, which reduces the influence of more distant causal dependencies in higher-order gradient computation.
• The method uses a hyperparameter τ to control the relative weighting between Monte Carlo samples and value function estimates, enabling explicit bias-variance trade-offs.
• The framework is fully compatible with automatic differentiation, requiring minimal code changes—only a few lines added to existing codebases using higher-order gradients.
• The approach generalizes the DiCE objective to support higher-order derivatives while maintaining theoretical correctness and practical usability.

Experimental results

Research questions

• RQ1 Can a single objective be derived that enables low-variance, unbiased estimation of any-order derivatives in stochastic computation graphs with Markov structure?
• RQ2 To what extent can arbitrary advantage estimators be integrated into higher-order gradient estimation to control bias and variance?
• RQ3 How does causal discounting via λ affect the bias-variance trade-off in higher-order derivatives?
• RQ4 Can the proposed method improve training stability and performance in meta-RL and continuous control tasks?
• RQ5 What is the impact of hyperparameters τ and λ on convergence and final performance in real-world RL applications?

Key findings

• In small random MDPs with analytic solutions, Loaded DiCE produces unbiased and low-variance higher-order derivative estimates when using a perfect value function.
• The hyperparameters τ and λ enable effective and flexible trade-offs between bias and variance in higher-order gradient estimation, with τ controlling the balance between Monte Carlo and value-based estimates.
• In meta-RL on CheetahDir and CheetahVel tasks, a non-zero λ (e.g., λ=0.5) led to faster learning than the unbiased λ=1.0 version, though both converged to similar performance levels.
• High τ values increased variance and degraded performance in short-horizon tasks, indicating that τ should be tuned based on return variance and value function quality.
• The method demonstrated significant impact in meta-RL, showing that controlling higher-order estimator bias and variance leads to measurable improvements in training efficiency and final policy performance.
• The framework requires only minimal code changes—just a few lines—making it readily deployable in existing higher-order RL codebases.