[Paper Review] Worst Cases Policy Gradients

Yichuan Charlie Tang, Jian Zhang+1 more|arXiv (Cornell University)|Nov 9, 2019

Reinforcement Learning in Robotics47 references34 citations

TL;DR

WCPG introduces a risk-sensitive actor-critic framework that models the distribution of future returns and optimizes CVaR for varying risk levels, enabling alpha-conditioned policies for safer, more robust control in driving simulations.

ABSTRACT

Recent advances in deep reinforcement learning have demonstrated the capability of learning complex control policies from many types of environments. When learning policies for safety-critical applications, it is essential to be sensitive to risks and avoid catastrophic events. Towards this goal, we propose an actor-critic framework that models the uncertainty of the future and simultaneously learns a policy based on that uncertainty model. Specifically, given a distribution of the future return for any state and action, we optimize policies for varying levels of conditional Value-at-Risk. The learned policy can map the same state to different actions depending on the propensity for risk. We demonstrate the effectiveness of our approach in the domain of driving simulations, where we learn maneuvers in two scenarios. Our learned controller can dynamically select actions along a continuous axis, where safe and conservative behaviors are found at one end while riskier behaviors are found at the other. Finally, when testing with very different simulation parameters, our risk-averse policies generalize significantly better compared to other reinforcement learning approaches.

Motivation & Objective

• Motivate safe, robust sequential decision making in safety-critical domains where outcomes are stochastic.
• Model the distribution of future returns and optimize a risk-aware criterion rather than only the mean return.
• Develop a scalable actor-critic architecture that supports continuous action spaces and off-policy learning.
• Enable a continuum of risk sensitivities by conditioning policies on a risk parameter alpha.

Proposed method

• Extend the DDPG actor-critic framework with a distributional critic that predicts both the mean Q and the variance Upsilon of future returns for each state-action pair.
• Model the return distribution Z(s,a) as Gaussian with mean Q and variance Upsilon and derive closed-form CVaR_alpha from these parameters (Gamma = Q - (phi(alpha)/Phi(alpha)) * sqrt(Upsilon).
• Compute CVaR in closed form to provide a risk-sensitive objective without costly sampling.
• Train a single neural network that takes alpha as an input, enabling a continuous family of risk-sensitive policies pi_theta(a|s, alpha).
• Use an off-policy experience replay buffer and backpropagate CVaR-based gradients through the critic to update both actor and critic networks.
• Discretize or parameterize alpha by feeding it as an input; during training, alpha is sampled per episode to learn a range of risk-tolerant policies.

Experimental results

Research questions

• RQ1 Does optimizing CVaR(alpha) via a distributional critic improve robustness to risk in continuous control tasks?
• RQ2 Can a single network learn a continuum of risk-sensitive policies by conditioning on alpha, and how does this affect performance across varying risk appetites?
• RQ3 How does the proposed approach compare to state-of-the-art RL methods when facing out-of-distribution environmental parameters?
• RQ4 What does the learned critic reveal about uncertainty and risk as the agent approaches critical maneuvers?
• RQ5 Can the approach transfer to related simulation environments (e.g., CARLA) with limited retraining?

Key findings

• WCPG learns risk-averse policies that reduce collision likelihood as alpha decreases.
• Smaller alpha leads to more conservative behavior and longer time-to-completion in driving tasks.
• The critic’s estimated uncertainty increases in high-risk situations (e.g., near oncoming traffic) and decreases after maneuvers are completed.
• Under extrapolation to out-of-distribution environment parameters, WCPG with low alpha shows superior robustness compared to baseline methods.
• Policies trained with WCPG exhibit better generalization to different simulation parameters and transfer to CARLA scenarios, maintaining robustness with lower alpha.