[Paper Review] The problem with DDPG: understanding failures in deterministic environments with sparse rewards

Guillaume Matheron, Nicolas Perrin+1 more|arXiv (Cornell University)|Nov 26, 2019

Reinforcement Learning in RoboticsComputer Science25 references61 citations

TL;DR

The paper formalizes why deterministic actor-critic methods like DDPG can fail in simple, sparse-reward environments, causing deadlock cycles where rewards are found but not exploited, and proposes potential remedies.

ABSTRACT

In environments with continuous state and action spaces, state-of-the-art actor-critic reinforcement learning algorithms can solve very complex problems, yet can also fail in environments that seem trivial, but the reason for such failures is still poorly understood. In this paper, we contribute a formal explanation of these failures in the particular case of sparse reward and deterministic environments. First, using a very elementary control problem, we illustrate that the learning process can get stuck into a fixed point corresponding to a poor solution. Then, generalizing from the studied example, we provide a detailed analysis of the underlying mechanisms which results in a new understanding of one of the convergence regimes of these algorithms. The resulting perspective casts a new light on already existing solutions to the issues we have highlighted, and suggests other potential approaches.

Motivation & Objective

• Explain how DDPG can fail in deterministic, sparse-reward environments using a simple 1D toy problem.
• Analyze the mechanisms leading to a deadlock where neither actor nor critic evolves after initial failures to exploit rewards.
• Generalize the identified failure mode to broader continuous-action actor-critic algorithms.
• Explore potential solutions and practical implications to mitigate cyclic convergence in such settings.

Proposed method

• Introduce a simple 1D toy environment with continuous state and action spaces and a sparse reward function to study failures of DDPG.
• Analyze the learning dynamics and identify a deadlock cycle where the actor converges to a saturated policy and the critic fails to propagate reward information.
• Provide formal arguments and proofs (with simplified assumptions) showing how Q converges toward Q^π and becomes piecewise-constant, leading to vanishing gradients for the actor.
• Demonstrate that certain updates (using Q′(s′,π(s′)) in the critic) and the deterministic policy gradient can trap the agent in poor policies.
• Compare traditional DDPG with alternatives like ddpg-argmax and SAC to illustrate how removing the deterministic max operator or introducing stochasticity helps avoid the deadlock.
• Discuss the impact of function approximation and how it interacts with the identified failure mode.

Experimental results

Research questions

• RQ1 What failure modes arise for deterministic policy-gradient updates in continuous-action, sparse-reward environments?
• RQ2 How does the interaction between the critic update target (Q′(s′,π(s′))) and the deterministic actor update contribute to deadlock?
• RQ3 Can alternative algorithms (e.g., stochastic actors, explicit max over actions, or auxiliary tasks) mitigate the observed failures in simple benchmarks and in sparse-reward variants of continuous-control tasks?
• RQ4 To what extent do function approximation and over/under-estimation bias influence the cyclic convergence mechanism?
• RQ5 Do these failure modes generalize beyond the 1D toy to more complex environments like sparse Reacher-v2 or HalfCheetah-v2?

Key findings

• DDPG can fail on an extremely simple 1D toy task with sparse rewards, achieving less than 100% success across seeds.
• The agent can enter a deadlock where neither actor nor critic update effectively propagates rewards, even when rewards are encountered.
• The critic tends toward a piecewise-constant function Q^π, causing near-zero gradients at the actor’s current policy and stalling policy improvement.
• Early discovery of reward strongly correlates with successful convergence to the optimal policy; late reward discovery increases failure likelihood.
• Replacing the deterministic max-actor update or using stochastic policies (as in SAC) can resolve the deadlock by avoiding reliance on Q(s′,π(s′)) in the critic/actor updates.
• Function approximators can both amplify and mitigate the issue, due to smoothing of discontinuities and introduction of local extrema.