[Paper Review] Hyperbolic Discounting and Learning over Multiple Horizons

William Fedus, Carles Gelada+3 more|arXiv (Cornell University)|Feb 19, 2019

Mathematical and Theoretical Analysis67 references52 citations

TL;DR

The paper introduces a practical method to implement hyperbolic (non-exponential) discounting in reinforcement learning by aggregating many exponentially discounted Q-values over multiple horizons, and shows that multi-horizon learning also improves performance as an auxiliary task.

ABSTRACT

Reinforcement learning (RL) typically defines a discount factor as part of the Markov Decision Process. The discount factor values future rewards by an exponential scheme that leads to theoretical convergence guarantees of the Bellman equation. However, evidence from psychology, economics and neuroscience suggests that humans and animals instead have hyperbolic time-preferences. In this work we revisit the fundamentals of discounting in RL and bridge this disconnect by implementing an RL agent that acts via hyperbolic discounting. We demonstrate that a simple approach approximates hyperbolic discount functions while still using familiar temporal-difference learning techniques in RL. Additionally, and independent of hyperbolic discounting, we make a surprising discovery that simultaneously learning value functions over multiple time-horizons is an effective auxiliary task which often improves over a strong value-based RL agent, Rainbow.

Motivation & Objective

• Question the use of a single exponential discount in RL and motivate time-preference models aligned with empirical hyperbolic discounts.
• Show that hyperbolic discounting can be approximated within TD-learning using an integral over exponential discounts.
• Demonstrate a practical, deep-learning approach to compute hyperbolic Q-values via multi-horizon Q-functions.
• Investigate hazard-based interpretations and equivalences between hazard rate priors and discount functions.
• Assess the potential of multi-horizon auxiliary tasks to improve baseline RL agents in complex environments.

Proposed method

• Formalize hazard and discount function equivalence to justify discounting as robustness to risk.
• Derive hyperbolic Q-values as an integral over exponential Q-values across a continuum of gamma values.
• Propose a practical approximation using a finite set of gamma values with Riemann-sum-like weighting.
• Use a deep network to learn multiple Q-values sharing parameters but discounted with different gammas.
• Establish the exponential-weighting condition to generalize beyond hyperbolic discounts.
• Apply the approach in Pathworld and the ALE to evaluate performance gains and auxiliary-task benefits.

Experimental results

Research questions

• RQ1 Can hyperbolic and other non-exponential discounts be computed from standard TD-learning by aggregating exponentially discounted values?
• RQ2 Does learning multiple Q-values across different horizons act as a beneficial auxiliary task beyond strong baselines like Rainbow?
• RQ3 Under hazard uncertainty or non-trivial intertemporal trade-offs, when is hyperbolic discounting advantageous?
• RQ4 What is the equivalence between hazard priors and discount functions in MDPs, and how can this guide robust policy learning?
• RQ5 How well does the finite-horizon gamma approximation capture hyperbolic discounting in high-dimensional RL domains?

Key findings

• Hyperbolic discounting can be computed as an integral over exponential discounts, enabling TD methods to approximate non-exponential preferences.
• A finite set of exponentially discounted Q-values, combined with appropriate weights, can approximate hyperbolic Q-values in practice.
• Learning multiple Q-values over different horizons can serve as an effective auxiliary task, improving performance over strong baselines in the ALE.
• The Pathworld environment demonstrates that hyperbolic discounting is beneficial under hazard uncertainty and non-trivial intertemporal choices.
• Hazard priors correspond to specific discount functions, providing a principled link between risk modeling and discounting in RL.
• The approach yields robust policies when environments exhibit uncertain hazards and risk of reward realization.