[Paper Review] An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient

Yudong Luo, Guiliang Liu+2 more|arXiv (Cornell University)|Jul 17, 2023

Health Systems, Economic Evaluations, Quality of LifeEconomics, Econometrics and Finance3 citations

TL;DR

This paper proposes Gini deviation (GD) as a superior alternative to variance for risk-averse policy gradient reinforcement learning. By deriving a policy gradient algorithm from GD's quantile representation, the method achieves higher returns with lower risk—outperforming variance-based approaches in learning stability and robustness, especially in domains where variance-based methods fail to converge.

ABSTRACT

Restricting the variance of a policy's return is a popular choice in risk-averse Reinforcement Learning (RL) due to its clear mathematical definition and easy interpretability. Traditional methods directly restrict the total return variance. Recent methods restrict the per-step reward variance as a proxy. We thoroughly examine the limitations of these variance-based methods, such as sensitivity to numerical scale and hindering of policy learning, and propose to use an alternative risk measure, Gini deviation, as a substitute. We study various properties of this new risk measure and derive a policy gradient algorithm to minimize it. Empirical evaluation in domains where risk-aversion can be clearly defined, shows that our algorithm can mitigate the limitations of variance-based risk measures and achieves high return with low risk in terms of variance and Gini deviation when others fail to learn a reasonable policy.

Motivation & Objective

• To address the limitations of variance-based risk measures in mean-variance reinforcement learning, such as sensitivity to reward scaling and interference with policy learning.
• To propose Gini deviation (GD) as a more robust dispersion measure that better captures risk without distorting reward signals.
• To develop a practical policy gradient algorithm that minimizes GD using a quantile-based gradient estimator.
• To empirically validate that GD-based learning achieves higher returns with lower risk than variance-based methods in challenging environments.
• To demonstrate the method's superiority in risk-averse domains where variance-based approaches fail to learn stable policies.

Proposed method

• Proposes Gini deviation (GD) as a new risk measure that quantifies dispersion in returns using the Gini mean difference, offering better interpretability and robustness than variance.
• Derives a policy gradient update from the quantile representation of GD to enable differentiable optimization, avoiding the need for double sampling or complex reward modifications.
• Uses a sampling-based estimator to compute gradients of GD, enabling end-to-end training with standard off-policy or on-policy RL frameworks.
• Applies the GD minimization objective within a policy gradient framework, maintaining the same policy architecture and training procedure as standard RL, but with an added risk regularization term.
• Employs a modified policy update rule that balances expected return and GD, using a hyperparameter to control risk-aversion trade-offs.
• Validates the method in MuJoCo, Lunar Lander, and custom risk-averse environments, comparing against variance-based and risk-neutral baselines.

Experimental results

Research questions

• RQ1 Can Gini deviation serve as a more robust and effective risk measure than variance in mean-variance reinforcement learning?
• RQ2 Does using GD instead of variance improve policy learning stability and convergence in risk-averse RL settings?
• RQ3 How does the proposed GD-based policy gradient method compare to variance-based methods in terms of return and risk (measured by both variance and GD)?
• RQ4 In environments where variance-based methods fail to learn, can the GD-based method still discover high-return, low-risk policies?
• RQ5 How sensitive is the GD-based method to reward scaling and reward shaping compared to variance-based approaches?

Key findings

• The proposed Gini deviation-based method successfully learns risk-averse policies in environments where variance-based methods fail to converge or produce unstable behavior.
• In location-based risky domains (e.g., Guarded Maze), the method achieves a lower rate of visiting noisy regions (e.g., 10% vs. 20% for variance-based methods) while maintaining higher expected returns.
• In distance-based risky domains (e.g., HalfCheetah and Swimmer with decaying noisy rewards), only the GD-based method (MG) shows a consistent forward movement tendency, indicating superior risk-aware behavior.
• When other methods fail to learn a reasonable policy, the GD-based method achieves significantly higher returns and lower risk—measured by both variance and Gini deviation—than risk-neutral or variance-based baselines.
• The GD-based method maintains lower return variance and Gini deviation than variance-based methods across all evaluated environments, especially in complex MuJoCo tasks.
• The method demonstrates robustness to reward scaling and avoids the policy degradation caused by reward modification strategies used in variance-based approaches.