[Paper Review] STAR-RT: Visual attention for real-time video game playing

Iuliia Kotseruba, John K. Tsotsos|arXiv (Cornell University)|Nov 26, 2017

Digital Image Processing Techniques9 references3 citations

TL;DR

STAR-RT presents the first real-time implementation of the STAR cognitive architecture, using visual attention and Cognitive Programs to control a vision system for playing closed-source video games. It achieves human-level performance in real time using only visual input, demonstrating the framework's viability for complex, dynamic visual tasks.

ABSTRACT

In this paper we present STAR-RT - the first working prototype of Selective Tuning Attention Reference (STAR) model and Cognitive Programs (CPs). The Selective Tuning (ST) model received substantial support through psychological and neurophysiological experiments. The STAR framework expands ST and applies it to practical visual tasks. In order to do so, similarly to many cognitive architectures, STAR combines the visual hierarchy (based on ST) with the executive controller, working and short-term memory components and fixation controller. CPs in turn enable the communication among all these elements for visual task execution. To test the relevance of the system in a realistic context, we implemented the necessary components of STAR and designed CPs for playing two closed-source video games - Canabaltand Robot Unicorn Attack. Since both games run in a browser window, our algorithm has the same amount of information and the same amount of time to react to the events on the screen as a human player would. STAR-RT plays both games in real time using only visual input and achieves scores comparable to human expert players. It thus provides an existence proof for the utility of the particular CP structure and primitives used and the potential for continued experimentation and verification of their utility in broader scenarios.

Motivation & Objective

• To develop a biologically plausible, real-time visual attention system capable of controlling complex vision tasks in dynamic environments.
• To validate the STAR framework—combining Selective Tuning (ST) and Cognitive Programs (CPs)—in a practical, real-world application.
• To demonstrate that a vision system using only visual input can achieve performance comparable to expert human players in real-time video games.
• To bridge the gap between theoretical models of visual attention and practical, real-time AI systems for dynamic, visually rich tasks.
• To provide a testbed for future research on visual attention, working memory, and task-directed processing in vision systems.

Proposed method

• Adapted the Selective Tuning (ST) model of visual attention to support both top-down and bottom-up processing in a hierarchical visual system.
• Designed Cognitive Programs (CPs) as executable, task-specific routines that coordinate the visual hierarchy, working memory, and fixation controller.
• Implemented a dual-memory architecture with task working memory (tWM) and visual working memory (vWM) to store and manage object and platform locations.
• Used external functions to compute frame-to-frame displacements, match objects across frames, and filter false positives in dynamic scenes.
• Applied foveation and fixation control to simulate human-like attention shifts, focusing processing on relevant regions of interest.
• Integrated the system into a single-threaded pipeline to manage real-time constraints, processing each frame independently while maintaining state across frames.

Experimental results

Research questions

• RQ1 Can the STAR cognitive architecture, with its combination of visual attention and task-directed Cognitive Programs, enable real-time performance in complex video games?
• RQ2 How effective is the integration of selective attention, working memory, and executive control in enabling autonomous game play without prior game engine access?
• RQ3 To what extent does the system's performance match that of human expert players when using only visual input?
• RQ4 What role do working memory and attention modulation play in maintaining performance under dynamic visual conditions?
• RQ5 How do visual artifacts from real camera feeds (e.g., motion blur, lens distortion) affect system robustness and performance?

Key findings

• STAR-RT successfully plays Canabalt and Robot Unicorn Attack in real time using only visual input, achieving scores comparable to human expert players.
• The system maintains real-time performance by processing each frame independently and using a compact, stateful representation in visual and task working memory.
• The removal of saliency-based attention (e.g., AIM) significantly increases processing load and error rates, demonstrating its critical role in filtering input.
• The framework effectively handles dynamic stimuli through frame differencing, object matching, and false positive rejection, even under motion blur and lens distortion.
• The system exhibits behavior resembling human fixation patterns, suggesting plausible mimicry of human visual attention mechanisms.
• Camera-based input introduces significant challenges such as ghosting, motion blur, and Moiré patterns, which degrade performance compared to screenshot-based input.