[Paper Review] Mask Atari for Deep Reinforcement Learning as POMDP Benchmarks

Yang Shao, Quan Kong+4 more|arXiv (Cornell University)|Mar 31, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

Mask Atari introduces a new DRL benchmark based on Atari 2600 games with controllable, moveable masks to simulate field-of-view limitations and enable Active Information Gathering (AIG) in partially observable Markov decision processes (POMDPs). The benchmark isolates cognitive performance by modeling human-like receptive fields, and experiments show that auxiliary rewards for information gathering accelerate learning but can lead to sub-optimal policies.

ABSTRACT

We present Mask Atari, a new benchmark to help solve partially observable Markov decision process (POMDP) problems with Deep Reinforcement Learning (DRL)-based approaches. To achieve a simulation environment for the POMDP problems, Mask Atari is constructed based on Atari 2600 games with controllable, moveable, and learnable masks as the observation area for the target agent, especially with the active information gathering (AIG) setting in POMDPs. Given that one does not yet exist, Mask Atari provides a challenging, efficient benchmark for evaluating the methods that focus on the above problem. Moreover, the mask operation is a trial for introducing the receptive field in the human vision system into a simulation environment for an agent, which means the evaluations are not biased from the sensing ability and purely focus on the cognitive performance of the methods when compared with the human baseline. We describe the challenges and features of our benchmark and evaluate several baselines with Mask Atari.

Motivation & Objective

• To address the lack of efficient, vision-rich POMDP benchmarks focused on Active Information Gathering (AIG) in complex environments.
• To simulate human-like receptive fields in reinforcement learning by introducing movable, learnable masks as the agent's observation window.
• To provide a benchmark that isolates cognitive performance from sensing limitations, enabling fair comparison with human baselines.
• To evaluate the effectiveness of DRL models and heuristic AIG strategies in partially observable, high-dimensional visual environments.
• To enable ablation studies on mask parameters (size, speed, number) to understand their impact on learning performance.

Proposed method

• Constructed Mask Atari by modifying Atari 2600 games with a controllable mask that defines the agent's field of view.
• Enabled dynamic mask movement via additional action space, allowing the agent to actively explore the environment.
• Designed a mask interface that supports multiple masks with customizable size, speed, and position.
• Introduced auxiliary reward functions to encourage information-gathering behavior: one based on image novelty and another on mask coverage expansion.
• Used standard DRL algorithms (e.g., A2C-CNN) to train agents and evaluate performance across multiple games.
• Performed ablation studies on mask scale, speed, and number to analyze their impact on learning and final scores.

Experimental results

Research questions

• RQ1 How does the size of the mask affect learning performance and final scores in vision-rich POMDP environments?
• RQ2 What is the impact of mask movement speed on the difficulty of the AIG task and agent performance?
• RQ3 How does increasing the number of masks influence the trade-off between information access and action space complexity?
• RQ4 Can auxiliary rewards based on information-gathering behavior accelerate learning in DRL agents on Mask Atari?
• RQ5 To what extent does the internal structure of the AIG setting enable faster learning without compromising final policy quality?

Key findings

• Increasing mask size from 50 to 100 pixels improved performance in most games, with a notable 20% increase in average score on MsPacman.
• Slower mask movement (10 px/frame) generally made games more challenging, but the effect was less pronounced than changes in mask size.
• Increasing mask speed from 10 to 50 px/frame had a milder impact on performance compared to mask size, suggesting speed is less critical than field-of-view extent.
• Using two masks led to mixed results: half the games saw higher scores, while the other half performed worse, indicating a trade-off between expanded visibility and increased action space complexity.
• The auxiliary reward for seeking new information accelerated learning in Asterix, Breakout, and MsPacman, but resulted in sub-optimal final policies.
• The auxiliary reward for expanding field of view similarly accelerated learning but also led to sub-optimal performance, indicating that AIG-based rewards can speed up training at the cost of final policy quality.