[Paper Review] Munchausen Reinforcement Learning

Nino Vieillard, Olivier Pietquin+1 more|arXiv (Cornell University)|Jul 28, 2020

Experimental Behavioral Economics StudiesSocial Sciences37 citations

TL;DR

The paper introduces Munchausen Reinforcement Learning (M-RL), which augments rewards with the scaled log-policy to bootstrap learning; it yields M-DQN and M-IQN that outperform non-distributional and distributional baselines on Atari, with theoretical insights on implicit KL regularization and action-gap increase.

ABSTRACT

Bootstrapping is a core mechanism in Reinforcement Learning (RL). Most algorithms, based on temporal differences, replace the true value of a transiting state by their current estimate of this value. Yet, another estimate could be leveraged to bootstrap RL: the current policy. Our core contribution stands in a very simple idea: adding the scaled log-policy to the immediate reward. We show that slightly modifying Deep Q-Network (DQN) in that way provides an agent that is competitive with distributional methods on Atari games, without making use of distributional RL, n-step returns or prioritized replay. To demonstrate the versatility of this idea, we also use it together with an Implicit Quantile Network (IQN). The resulting agent outperforms Rainbow on Atari, installing a new State of the Art with very little modifications to the original algorithm. To add to this empirical study, we provide strong theoretical insights on what happens under the hood -- implicit Kullback-Leibler regularization and increase of the action-gap.

Motivation & Objective

• Motivate leveraging the current policy as a bootstrap signal in TD learning.
• Propose a simple, generic Munchausen update that adds the scaled log-policy to rewards.
• Demonstrate the method's effectiveness by applying it to DQN and IQN and comparing to strong baselines.
• Provide theoretical interpretation via implicit KL regularization and action-gap analysis.

Proposed method

• Augment the TD target with alpha times tau times log pi(a|s) added to the reward.
• Generalize DQN to maximum-entropy RL and then introduce the Munchausen term to obtain M-DQN.
• Apply the Munchausen idea to IQN to create M-IQN and show state-of-the-art performance on Atari.
• Rewrite M-DQN in an abstract dynamic programming framework to reveal implicit KL regularization between consecutive policies.
• Analyze action-gap amplification and connect M-RL to CVI, DPP, and AL.
• Conduct empirical ablations to isolate the Munchausen contribution and assess robustness of hyperparameters alpha, tau, and log-policy clipping.

Experimental results

Research questions

• RQ1 Does augmenting rewards with scaled log-policy improve sample efficiency and performance over standard TD methods?
• RQ2 Can Munchausen RL provide competitive or superior performance without distributional RL tricks on Atari?
• RQ3 What theoretical guarantees or interpretations (e.g., KL regularization, action-gap increase) accompany the Munchausen updates?
• RQ4 How does Munchausen RL extend to distributional RL settings (e.g., IQN) and affect state-of-the-art baselines?

Key findings

• M-DQN significantly outperforms DQN and C51 on Atari across 60 games using a simple modification to the regression target.
• M-IQN surpasses Rainbow, achieving new state-of-the-art results among non-distributed baselines and competitive with distRL agents.
• M-DQN and M-IQN benefit from an implicit KL regularization effect between successive policies, improving stability and error averaging.
• The Munchausen framework increases the action-gap in a quantifiable way, contributing to robustness against approximation errors.
• Ablation studies show Munchausen updates outperform Soft-DQN, AL, and Adam DQN, highlighting the value of the Munchausen term beyond entropy regularization alone.
• Theoretical connections link M-VI to MD-VI and CVI/DPP/AL literature, with bounds that explain improved convergence properties in the presence of function approximation.