[Paper Review] Psychlab: A Psychology Laboratory for Deep Reinforcement Learning Agents

Joel Z. Leibo, Cyprien de Masson d’Autume+11 more|arXiv (Cornell University)|Jan 24, 2018

EEG and Brain-Computer Interfaces14 references61 citations

TL;DR

Psychlab provides a DeepMind Lab-based platform to run classic cognitive psychology experiments with both humans and deep RL agents, enabling direct performance comparisons and psychophysics analyses. A case study on UNREAL reveals size/contrast effects leading to a foveal vision improvement.

ABSTRACT

Psychlab is a simulated psychology laboratory inside the first-person 3D game world of DeepMind Lab (Beattie et al. 2016). Psychlab enables implementations of classical laboratory psychological experiments so that they work with both human and artificial agents. Psychlab has a simple and flexible API that enables users to easily create their own tasks. As examples, we are releasing Psychlab implementations of several classical experimental paradigms including visual search, change detection, random dot motion discrimination, and multiple object tracking. We also contribute a study of the visual psychophysics of a specific state-of-the-art deep reinforcement learning agent: UNREAL (Jaderberg et al. 2016). This study leads to the surprising conclusion that UNREAL learns more quickly about larger target stimuli than it does about smaller stimuli. In turn, this insight motivates a specific improvement in the form of a simple model of foveal vision that turns out to significantly boost UNREAL's performance, both on Psychlab tasks, and on standard DeepMind Lab tasks. By open-sourcing Psychlab we hope to facilitate a range of future such studies that simultaneously advance deep reinforcement learning and improve its links with cognitive science.

Motivation & Objective

• Introduce Psychlab as a flexible psychology laboratory inside DeepMind Lab for testing humans and deep RL agents on classical cognitive psychology tasks.
• Provide implementations of tasks like visual search, change detection, random dot motion discrimination, and multiple object tracking.
• Analyze visual psychophysics of a state-of-the-art deep RL agent (UNREAL) and derive potential agent enhancements.
• Open-source Psychlab to foster future studies linking cognitive science and deep reinforcement learning.

Proposed method

• Describe the Psychlab platform and its DM-Lab integration.
• Outline the Lua-based Psychlab API for building stimuli and tasks on the monitor with widgets and gaze callbacks.
• Explain the reinforcement learning setup and reward scheme used in Psychlab experiments.
• Present adaptive staircase procedures to measure psychophysical thresholds and enable curriculum-like learning for agents.
• Detail a case study applying visual psychophysics to UNREAL, including acuity, contrast sensitivity, and size/contrast effects.

Experimental results

Research questions

• RQ1 How do deep RL agents perform on classical psychophysics tasks compared to humans within the Psychlab environment?
• RQ2 What cognitive/perceptual faculties can be isolated using Psychlab tasks and how do agents differ from humans across these tasks?
• RQ3 Can psychophysical measurements (thresholds, psychometric functions) reveal limitations or strengths of deep RL agents?
• RQ4 Do simple architectural or input-processing changes (e.g., foveal vision models) improve agent performance on Psychlab tasks and related DM-Lab tasks?

Key findings

• UNREAL shows worse visual acuity than humans even at the same input resolution.
• UNREAL's contrast sensitivity differs from humans, performing worse at moderate contrasts but better at very low contrasts.
• UNREAL is strongly influenced by target size and presence of lures in pointing tasks, suggesting weight-sharing effects in convolutional nets accelerate learning for larger targets.
• Glass pattern detection reveals human-like thresholds in UNREAL, with similar sigmoid psychometric shapes and 75% threshold around 0.5 coherence for both white and black patterns.
• Introducing a simple foveal vision mechanism improves UNREAL performance on Psychlab and standard DM-Lab tasks.
• Psychlab enables identification of learning dynamics and potential suboptimal local optima arising from convolutional weight sharing in deep RL agents.