[Paper Review] Deep Exploration via Randomized Value Functions

Ian Osband, Benjamin Van Roy+2 more|arXiv (Cornell University)|Mar 22, 2017

Advanced Bandit Algorithms Research84 references68 citations

TL;DR

The paper proposes a deep RL approach that integrates randomized value functions to drive deep exploration, leveraging a Markov planning-style update with a prior over parameters and a memory buffer.

ABSTRACT

We study the use of randomized value functions to guide deep exploration in reinforcement learning. This offers an elegant means for synthesizing statistically and computationally efficient exploration with common practical approaches to value function learning. We present several reinforcement learning algorithms that leverage randomized value functions and demonstrate their efficacy through computational studies. We also prove a regret bound that establishes statistical efficiency with a tabular representation.

Motivation & Objective

• Motivate efficient exploration in deep reinforcement learning through randomized value function representations.
• Define a learning framework that combines a TD loss, a prior over parameters, and a replay buffer to guide exploration.
• Introduce an update procedure that generates a perturbed value function estimate for planning.

Proposed method

• Use randomized value functions with a TD error-based loss to update estimates.
• Maintain a memory buffer (buffer) of past observations for training.
• Incorporate a prior distribution over parameters (prior) to regularize learning.
• Define a planning horizon H that governs lookahead during updates.
• Compute a perturbed/updated value function estimate tilde{θ} as part of the update sequence.

Experimental results

Research questions

• RQ1 Can randomized value functions improve deep exploration efficiency in reinforcement learning?
• RQ2 How should a prior over parameters and a replay buffer interact to stabilize deep exploration?
• RQ3 What role does the planning horizon play in the effectiveness of the randomized value function approach?

Key findings

• The approach introduces a structured update involving a prior, buffer data, and planning to produce a perturbed value function estimate.
• The method frames updates with a TD error loss, a regularization function, and a memory buffer, aiming to balance exploration and value estimation.
• The document outlines an algorithmic progression from prior mean and buffered data toward a new value function estimate.