[Paper Review] RUDDER: Return Decomposition for Delayed Rewards

Jose A. Arjona-Medina, Michael Gillhofer+4 more|arXiv (Cornell University)|Jun 20, 2018

Reinforcement Learning in Robotics121 references59 citations

TL;DR

RUDDER introduces reward redistribution and return decomposition to address delayed rewards, turning RL into a regression task with LSTM-based return decomposition, yielding significant speedups and improved Atari performance.

ABSTRACT

We propose RUDDER, a novel reinforcement learning approach for delayed rewards in finite Markov decision processes (MDPs). In MDPs the Q-values are equal to the expected immediate reward plus the expected future rewards. The latter are related to bias problems in temporal difference (TD) learning and to high variance problems in Monte Carlo (MC) learning. Both problems are even more severe when rewards are delayed. RUDDER aims at making the expected future rewards zero, which simplifies Q-value estimation to computing the mean of the immediate reward. We propose the following two new concepts to push the expected future rewards toward zero. (i) Reward redistribution that leads to return-equivalent decision processes with the same optimal policies and, when optimal, zero expected future rewards. (ii) Return decomposition via contribution analysis which transforms the reinforcement learning task into a regression task at which deep learning excels. On artificial tasks with delayed rewards, RUDDER is significantly faster than MC and exponentially faster than Monte Carlo Tree Search (MCTS), TD(λ), and reward shaping approaches. At Atari games, RUDDER on top of a Proximal Policy Optimization (PPO) baseline improves the scores, which is most prominent at games with delayed rewards. Source code is available at \url{https://github.com/ml-jku/rudder} and demonstration videos at \url{https://goo.gl/EQerZV}.

Motivation & Objective

• Address long-term credit assignment with delayed rewards in finite MDPs.
• Introduce reward redistribution to create return-equivalent SDPs with zero expected future rewards.
• Develop return decomposition to turn RL into a regression task for efficient learning.
• Leverage LSTM-based return decomposition to identify state-action contributions to returns.
• Demonstrate speedups over TD, MC, MCTS, and reward shaping on synthetic tasks and Atari games.

Proposed method

• Define return-equivalent sequence-Markov decision processes (SDPs) via reward redistribution.
• Aim for an optimal redistribution that yields zero expected future rewards, enabling Q-value estimation by the mean of immediate rewards.
• Use return decomposition to identify state-action contributions to the sequence return.
• Employ LSTM-based return decomposition to train a model predicting the sequence-wide return and derive redistributed rewards from prediction differences.
• Phase-based learning: safe exploration, lessons replay buffer, then LSTM-based return decomposition.
• Integrate redistributed rewards into Q-learning, policy gradient, or PPO-based frameworks (e.g., PPO with redistributed rewards).

Experimental results

Research questions

• RQ1 Can reward redistribution produce return-equivalent SDPs with zero expected future rewards while preserving optimal policies?
• RQ2 Does return decomposition via contribution analysis enable effective learning from delayed rewards using regression on complete episodes?
• RQ3 How does RUDDER performance compare to TD, MC, MCTS, and reward shaping on synthetic delayed-reward tasks and Atari games?
• RQ4 What are the practical benefits of using an LSTM for return decomposition over feedforward models in this setting?

Key findings

• RUDDER achieves significant speed-ups on artificial delayed-reward tasks compared to TD(λ), MC, MCTS, and reward shaping.
• Optimal reward redistribution theoretically yields zero expected future rewards, reducing Q-value estimation to the mean immediate reward.
• Return decomposition identifies the most contributing state-action pairs, enabling efficient redistribution of rewards.
• On Atari games, RUDDER improves PPO-based baselines, with strongest gains on games featuring delayed rewards.
• Experiments show substantial performance gains across finite-horizon MDPs with delayed rewards using the proposed LSTM-based approach.