[Paper Review] Behaviour Suite for Reinforcement Learning

Ian Osband, Yotam Doron+12 more|arXiv (Cornell University)|Aug 9, 2019

Reinforcement Learning in Robotics48 references36 citations

TL;DR

bsuite is a collection of targeted, scalable RL experiments and an open-source toolkit that evaluates and analyzes agent behavior to understand core RL capabilities. It provides principled diagnostics and reproducible analyses across codebases.

ABSTRACT

This paper introduces the Behaviour Suite for Reinforcement Learning, or bsuite for short. bsuite is a collection of carefully-designed experiments that investigate core capabilities of reinforcement learning (RL) agents with two objectives. First, to collect clear, informative and scalable problems that capture key issues in the design of general and efficient learning algorithms. Second, to study agent behaviour through their performance on these shared benchmarks. To complement this effort, we open source github.com/deepmind/bsuite, which automates evaluation and analysis of any agent on bsuite. This library facilitates reproducible and accessible research on the core issues in RL, and ultimately the design of superior learning algorithms. Our code is Python, and easy to use within existing projects. We include examples with OpenAI Baselines, Dopamine as well as new reference implementations. Going forward, we hope to incorporate more excellent experiments from the research community, and commit to a periodic review of bsuite from a committee of prominent researchers.

Motivation & Objective

• Provide clear, informative, scalable experiments that diagnose key RL capabilities.
• Enable reproducible evaluation and comparison across RL agents and codebases.
• Isolate and study fundamental RL issues such as exploration, memory, and credit assignment.

Proposed method

• Define a suite of diagnostic RL experiments with fixed environments, interaction regimes, and analysis pipelines.
• Score agent performance on each task with a [0,1] scale to enable quick comparisons.
• Offer open-source implementations, reference baselines, and automated analysis notebooks to facilitate reproducible research.
• Describe how bsuite experiments are designed to be targeted, simple, challenging, scalable, and fast.

Experimental results

Research questions

• RQ1 What core RL capabilities can be isolated and measured by targeted experiments?
• RQ2 How do different RL algorithms perform on diagnostic tasks that probe memory and exploration?
• RQ3 Can a common benchmarking library enable reproducible evaluation across different RL codebases?
• RQ4 What are the scaling properties of algorithms on diagnostic tasks as problem size increases?

Key findings

• Memory-length experiments show recurrent policies outperform feedforward ones for multi-step memory tasks, with clear scaling behavior.
• DQN and Bootstrapped DQN struggle with memory length beyond one step, while A2C shows strong performance up to a cutoff before randomness increases.
• Deep Sea exploration highlights the need for deep exploration, with Bootstrapped DQN providing superior scalability to larger problem sizes.
• .bsuite provides a quick, interpretable summary through a radar plot and a unified scoring mechanism across multiple experiments.
• Open-source tooling enables easy integration with existing RL codebases and facilitates reproducible analyses.