[Paper Review] Exploration by Distributional Reinforcement Learning

Yunhao Tang, Shipra Agrawal|arXiv (Cornell University)|May 4, 2018

Reinforcement Learning in Robotics16 references11 citations

TL;DR

This paper proposes a distributional reinforcement learning framework that unifies exploration methods by modeling return distributions with Gaussian parameters, enabling efficient exploration through parameter-space noise. The method, Gauss Exploration (GE), achieves superior performance on sparse-reward control tasks by balancing exploration and exploitation via learned uncertainty in value function parameters.

ABSTRACT

We propose a framework based on distributional reinforcement learning and recent attempts to combine Bayesian parameter updates with deep reinforcement learning. We show that our proposed framework conceptually unifies multiple previous methods in exploration. We also derive a practical algorithm that achieves efficient exploration on challenging control tasks.

Motivation & Objective

• To unify multiple existing exploration methods in deep reinforcement learning under a single distributional RL framework.
• To address the limitations of naive exploration strategies like ε-greedy in hard, sparse-reward environments.
• To develop a practical algorithm that enables systematic and efficient exploration through uncertainty in value function parameters.
• To demonstrate improved sample efficiency and performance on challenging control benchmarks compared to DQN, NoisyNet, and standard DQN with action noise.

Proposed method

• Models the return distribution using a parametric family of Gaussians, with parameters μθ and σθ for each network weight and bias.
• Applies a distributional Bellman operator to propagate return distributions, projecting to the closest parametric distribution using a discrepancy measure.
• Uses a hybrid objective combining expected Bellman error and entropy regularization to balance exploration and exploitation.
• Performs policy gradient updates through randomized critics, where the critic's parameters are sampled from a learned distribution to inject noise into the learning process.
• Employs a reparameterization trick to enable differentiable training of the parameter distribution, allowing end-to-end optimization of exploration behavior.
• Tunes hyperparameters ρ (controls parameter distribution spread) and σ (controls entropy regularization) via grid search to balance exploration and convergence.

Experimental results

Research questions

• RQ1 Can distributional reinforcement learning be used to unify existing exploration methods such as posterior sampling and parameter-space noise injection?
• RQ2 How does modeling the return distribution with a Gaussian family improve exploration in deep RL?
• RQ3 What is the impact of parameter distribution uncertainty on sample efficiency in sparse-reward environments?
• RQ4 How do the hyperparameters ρ and σ affect the trade-off between exploration and exploitation in the proposed framework?
• RQ5 Can the proposed method outperform standard DQN with ε-greedy and NoisyNet in challenging control tasks?

Key findings

• The proposed Gauss Exploration (GE) algorithm achieves more systematic and widespread state-space coverage compared to DQN, which tends to cluster in limited regions.
• On sparse-reward environments like Sparse CartPole, Sparse MountainCar, and Sparse Acrobot, GE significantly outperforms DQN and NoisyNet in learning progress and final performance.
• In continuous control tasks with sparse rewards, DDPG with randomized critics (using GE) makes progress at a higher rate than DDPG with action noise injection.
• Performance is non-monotonic in hyperparameters ρ and σ, indicating a critical trade-off between exploration breadth and convergence speed.
• The method successfully unifies posterior sampling for bandits and Bayesian parameter updates in DQN under a single distributional RL framework.
• Empirical results show that both ρ and σ are crucial for performance, with optimal values found via grid search on log scales.