[Paper Review] Noisy Networks for Exploration

Meire Fortunato, Mohammad Gheshlaghi Azar+10 more|arXiv (Cornell University)|Jun 30, 2017

Reinforcement Learning in Robotics44 references387 citations

TL;DR

NoisyNet injects learnable parametric noise into network weights to drive exploration, replacing standard exploration heuristics in DQN, Dueling, and A3C, yielding substantial gains across 57 Atari games.

ABSTRACT

We introduce NoisyNet, a deep reinforcement learning agent with parametric noise added to its weights, and show that the induced stochasticity of the agent's policy can be used to aid efficient exploration. The parameters of the noise are learned with gradient descent along with the remaining network weights. NoisyNet is straightforward to implement and adds little computational overhead. We find that replacing the conventional exploration heuristics for A3C, DQN and dueling agents (entropy reward and $ε$-greedy respectively) with NoisyNet yields substantially higher scores for a wide range of Atari games, in some cases advancing the agent from sub to super-human performance.

Motivation & Objective

• Motivate a noise-based exploration mechanism that couples exploration with learning through gradient-tuned weight perturbations.
• Replace traditional exploration strategies (epsilon-greedy, entropy bonuses) with NoisyNet in multiple deep RL architectures.
• Demonstrate performance improvements across a broad set of atari games and analyze how noise adapts during learning.

Proposed method

• Define noisy network layers where weights and biases are mu + sigma * epsilon with epsilon drawn from a fixed distribution.
• Use either independent or factorised Gaussian noise to generate epsilon, enabling scalable noise perturbations.
• Train both the network parameters and noise parameters via gradient descent, with Monte Carlo estimates of the loss over noisy networks.
• Apply NoisyNet to DQN, Dueling, and A3C by replacing corresponding linear layers with noisy layers and removing separate exploration tricks.
• Use NoisyNet-DQN and NoisyNet-Dueling with re-sampling of noise before every action; NoisyNet-A3C uses no entropy bonus and samples noise per rollout.
• Provide initialization schemes for mu and sigma of noisy parameters and detail the gradient computation for the NoisyNet loss.

Experimental results

Research questions

• RQ1 Does injecting learnable parameter noise into neural networks improve exploration efficiency in deep RL?
• RQ2 Can NoisyNet replace conventional exploration strategies across different deep RL algorithms (DQN, Dueling, A3C) with comparable or better performance?
• RQ3 How does the noise parameter evolve during training, and does it adapt to task difficulty or game—i.e., is exploration regime contextual?
• RQ4 What are the empirical gains in performance across a large set of Atari games when using NoisyNet variants?
• RQ5 Is the noise approach compatible with factorised noise to reduce computational overhead without sacrificing performance?

Key findings

• NoisyNet variants significantly improve mean and median human-normalised scores across 57 Atari games compared to baselines.
• DQN: mean improvement around 48% (median improvement 83 to 123 in human-normalised scores); NoisyNet-Dueling shows 30% median improvement (132 to 172).
• A3C with NoisyNet improves median human-normalised score by 18% (80 to 94).
• In several games (e.g., Beam Rider, Asteroids, Freeway), NoisyNet achieves super-human performance where baselines lagged.
• Factorised Gaussian noise in NoisyNet-A3C maintains performance while reducing computational overhead.
• NoisyNet often yields improvements throughout training rather than only at convergence, indicating enhanced exploration during learning.