[Paper Review] A learning gap between neuroscience and reinforcement learning

Samuel T. Wauthier, Pietro Mazzaglia+4 more|arXiv (Cornell University)|Apr 22, 2021

Neural dynamics and brain function32 references4 citations

TL;DR

This paper identifies a critical learning gap between neuroscience and reinforcement learning (RL) by demonstrating that state-of-the-art RL algorithms fail to solve a neuroscience-inspired T-maze environment designed to test belief-based reasoning and ambiguity resolution. The authors reimplement Friston et al.'s abstract T-maze as a pixel-based, game-like environment (E-maze) and show that even advanced algorithms like PPO, DQN, Rainbow, and Dreamer cannot reliably solve it, highlighting a need for RL benchmarks grounded in cognitive neuroscience principles to drive more biologically plausible intelligence.

ABSTRACT

Historically, artificial intelligence has drawn much inspiration from neuroscience to fuel advances in the field. However, current progress in reinforcement learning is largely focused on benchmark problems that fail to capture many of the aspects that are of interest in neuroscience today. We illustrate this point by extending a T-maze task from neuroscience for use with reinforcement learning algorithms, and show that state-of-the-art algorithms are not capable of solving this problem. Finally, we point out where insights from neuroscience could help explain some of the issues encountered.

Motivation & Objective

• To identify a disconnect between current reinforcement learning (RL) benchmarks and core cognitive challenges studied in neuroscience, such as ambiguity, memory, and belief formation.
• To demonstrate that state-of-the-art RL algorithms struggle with a T-maze environment designed to test belief-based decision-making, despite its simplicity in concept.
• To propose that future RL benchmarks should be grounded in well-established neuroscience tasks to foster algorithms inspired by biological cognition.
• To provide empirical evidence that current RL methods lack the representational and reasoning capabilities needed for tasks requiring persistent belief states.

Proposed method

• Reimplemented the abstract T-maze from Friston et al. (2016) as a pixel-based, game-like environment (E-maze) with continuous movement and visual observations.
• Defined the environment with a 198-degree field of view and a maximum of 250 moves to control sparsity and enable complex navigation.
• Used standard deep RL algorithms—DQN, Rainbow, PPO, and Dreamer—trained with standard hyperparameters adapted from the Atari 2600 benchmark.
• Applied intrinsic exploration bonuses via ICM and RND to encourage exploration in sparse-reward settings, adding them to the extrinsic reward signal.
• Preprocessed observations to 84×84 grayscale images, matching Atari benchmark standards, to ensure fair comparison with existing RL baselines.
• Trained agents using standard PPO with weight initialization and exploration scheduling, switching off intrinsic bonuses once 20% performance was reached.

Experimental results

Research questions

• RQ1 Can state-of-the-art deep reinforcement learning algorithms solve a neuroscience-inspired T-maze task that requires belief-based reasoning under ambiguity?
• RQ2 Why do current RL algorithms fail on this task despite their success on Atari and other game environments?
• RQ3 What role do intrinsic exploration methods like ICM and RND play in enabling learning in this ambiguous, sparse-reward environment?
• RQ4 To what extent do model-free and model-based RL approaches (e.g., PPO and Dreamer) succeed in resolving uncertainty through belief formation?
• RQ5 How can neuroscience-informed environments improve the development of more cognitively plausible RL algorithms?

Key findings

• None of the tested state-of-the-art RL algorithms—DQN, Rainbow, PPO, or Dreamer—achieved reliable performance on the E-maze environment, despite extensive hyperparameter tuning.
• Even with intrinsic exploration bonuses from ICM and RND, PPO agents failed to exceed 20% performance, indicating insufficient exploration in the belief-space.
• The environment's design, requiring persistent belief states and resolution of ambiguity via clues, exposed a fundamental limitation in current RL architectures.
• Model-free methods like PPO and DQN showed no significant improvement over random baseline, suggesting they lack mechanisms for maintaining and updating beliefs over time.
• The failure of Dreamer, a model-based agent, further indicates that world-model learning alone is insufficient without explicit mechanisms for belief representation.
• The results suggest that current RL algorithms do not naturally support the kind of probabilistic inference and memory-dependent reasoning observed in biological agents.