[Paper Review] Randomized Prior Functions for Deep Reinforcement Learning

Ian Osband, John Aslanides+1 more|arXiv (Cornell University)|Jun 8, 2018

Reinforcement Learning in RoboticsComputer Science6 references105 citations

TL;DR

The paper proposes randomized prior functions to enhance exploration in deep reinforcement learning within a standard agent–environment loop.

ABSTRACT

Dealing with uncertainty is essential for efficient reinforcement learning. There is a growing literature on uncertainty estimation for deep learning from fixed datasets, but many of the most popular approaches are poorly-suited to sequential decision problems. Other methods, such as bootstrap sampling, have no mechanism for uncertainty that does not come from the observed data. We highlight why this can be a crucial shortcoming and propose a simple remedy through addition of a randomized untrainable `prior' network to each ensemble member. We prove that this approach is efficient with linear representations, provide simple illustrations of its efficacy with nonlinear representations and show that this approach scales to large-scale problems far better than previous attempts.

Motivation & Objective

• Motivate the use of randomized priors to improve exploration in deep reinforcement learning.
• Describe how randomized prior functions integrate with a standard DRL training loop.
• Outline the agent–environment interaction workflow including replay buffer usage.

Proposed method

• Define an agent with methods act, update_buffer, learn_from_buffer.
• Run episodes where the agent learns from the buffer at each iteration.
• Reset the environment to obtain a new transition, then determine action via agent.act on the current state.
• Apply the action using environment.step and store the resulting transition with agent.update_buffer.
• Iterate over episodes to continually learn from the buffered transitions.

Experimental results

Research questions

• RQ1 Do randomized prior functions improve exploration efficiency in deep reinforcement learning?
• RQ2 What is the impact of randomized priors on learning stability and sample efficiency within the standard DRL training loop.

Key findings

• Not available in the provided excerpt.
• No quantitative results are shown in the supplied text.