[Paper Review] No-Regret Reinforcement Learning with Heavy-Tailed Rewards

Vincent Zhuang, Yanan Sui|arXiv (Cornell University)|Feb 25, 2021

Advanced Bandit Algorithms Research29 references4 citations

TL;DR

This paper proposes Heavy-UCRL2 and Heavy-Q-Learning, robust reinforcement learning algorithms designed for MDPs with heavy-tailed rewards (finite $(1+\epsilon)$-th moments). By leveraging robust mean estimation techniques like median-of-means, the algorithms achieve near-optimal regret bounds in the heavy-tailed regime, where reward estimation dominates learning difficulty. The approach generalizes to deep RL via Heavy-DQN, outperforming baselines on synthetic and standard benchmarks.

ABSTRACT

Reinforcement learning algorithms typically assume rewards to be sampled from light-tailed distributions, such as Gaussian or bounded. However, a wide variety of real-world systems generate rewards that follow heavy-tailed distributions. We consider such scenarios in the setting of undiscounted reinforcement learning. By constructing a lower bound, we show that the difficulty of learning heavy-tailed rewards asymptotically dominates the difficulty of learning transition probabilities. Leveraging techniques from robust mean estimation, we propose Heavy-UCRL2 and Heavy-Q-Learning, and show that they achieve near-optimal regret bounds in this setting. Our algorithms also naturally generalize to deep reinforcement learning applications; we instantiate Heavy-DQN as an example of this. We demonstrate that all of our algorithms outperform baselines on both synthetic MDPs and standard RL benchmarks.

Motivation & Objective

• To address the lack of theoretical and empirical studies on reinforcement learning with heavy-tailed reward distributions in MDPs.
• To demonstrate that learning heavy-tailed rewards becomes the dominant source of difficulty, surpassing transition probability learning.
• To design and analyze confidence-based RL algorithms—Heavy-UCRL2 and Heavy-Q-Learning—that achieve near-optimal regret under heavy-tailed rewards.
• To extend the framework to deep RL via Heavy-DQN, showing practical applicability.

Proposed method

• Uses robust mean estimation via median-of-means to handle rewards with only finite $(1+\epsilon)$-th moments.
• Replaces standard confidence intervals in UCRL2 and Q-learning with robust counterparts derived from concentration inequalities for heavy-tailed distributions.
• Constructs a lower bound proving that reward estimation difficulty dominates transition learning in the unbounded heavy-tailed setting.
• Employs a novel analysis framework to bound the regret of confidence-based algorithms under heavy-tailed rewards, incorporating robust variance and uncertainty terms.
• Applies the same robust estimation principles to deep Q-networks, resulting in Heavy-DQN with improved sample efficiency and stability.
• Validates theoretical claims through experiments on synthetic MDPs (SixArms, DoubleChain) and standard RL benchmarks.

Experimental results

Research questions

• RQ1 What is the fundamental difficulty in learning MDPs when rewards follow heavy-tailed distributions with only finite $(1+\epsilon)$-th moments?
• RQ2 Can robust mean estimation techniques be effectively adapted to confidence-based reinforcement learning algorithms like UCRL2 and Q-learning?
• RQ3 Does the difficulty of learning heavy-tailed rewards dominate that of learning transition dynamics in undiscounted MDPs?
• RQ4 Can robust RL algorithms achieve near-optimal regret bounds in the heavy-tailed setting?
• RQ5 How do robust algorithms generalize to deep reinforcement learning architectures?

Key findings

• The paper establishes a theoretical lower bound showing that learning heavy-tailed rewards asymptotically dominates the difficulty of learning transition probabilities in undiscounted MDPs.
• Heavy-UCRL2 and Heavy-Q-Learning achieve near-optimal regret bounds under the assumption of finite $(1+\epsilon)$-th moments, with explicit dependence on the tail index $\epsilon$.
• The regret bounds scale with $H^{3}SA\iota^{2}\sqrt{r_{\text{max}}^{3}}$ and $H^{\frac{1+3\epsilon}{\epsilon}}\sqrt{S^{3}A^{3}\iota^{4}\epsilon}$, reflecting the increased complexity of heavy-tailed estimation.
• Heavy-DQN, derived from the same robust estimation principles, outperforms standard DQN on both synthetic MDPs and standard RL benchmarks.
• Empirical results on SixArms and DoubleChain MDPs show consistent and significant performance gains for Heavy-UCRL2 and Heavy-Q-Learning over standard baselines.
• The algorithms demonstrate robustness to extreme reward outliers, maintaining stable learning even when rewards exhibit heavy-tailed behavior.