[Paper Review] Quantifying Human Behavior on the Block Design Test Through Automated Multi-Level Analysis of Overhead Video

Seung-Hwan Cha, James Ainooson+1 more|arXiv (Cornell University)|Nov 18, 2018

Spatial Cognition and Navigation6 citations

TL;DR

This paper presents an AI-driven framework that uses overhead RGB video and automated computer vision to quantitatively analyze human behavior during the Block Design Test (BDT), a standard neuropsychological assessment of visuospatial reasoning. The system, comprising the Automated Block Identification System (ABIS) and the Cognitive Analysis using Block Sequence (CABS) module, enables frame-level tracking of block placements and error detection, with CABS generating visualizations of performance metrics such as error counts and reaction times for clinical use.

ABSTRACT

The block design test is a standardized, widely used neuropsychological assessment of visuospatial reasoning that involves a person recreating a series of given designs out of a set of colored blocks. In current testing procedures, an expert neuropsychologist observes a person's accuracy and completion time as well as overall impressions of the person's problem-solving procedures, errors, etc., thus obtaining a holistic though subjective and often qualitative view of the person's cognitive processes. We propose a new framework that combines room sensors and AI techniques to augment the information available to neuropsychologists from block design and similar tabletop assessments. In particular, a ceiling-mounted camera captures an overhead view of the table surface. From this video, we demonstrate how automated classification using machine learning can produce a frame-level description of the state of the block task and the person's actions over the course of each test problem. We also show how a sequence-comparison algorithm can classify one individual's problem-solving strategy relative to a database of simulated strategies, and how these quantitative results can be visualized for use by neuropsychologists.

Motivation & Objective

• To address the lack of objective, real-time behavioral metrics in traditional Block Design Test (BDT) administration, which relies heavily on subjective neuropsychological judgment.
• To develop an automated system that captures detailed, frame-level behavioral data—such as block placements, errors, and movement sequences—during BDT performance.
• To enable neuropsychologists to access quantitative, visualizable summaries of problem-solving strategies and performance patterns using AI-derived metrics.
• To lay the foundation for scalable, objective assessment of visuospatial cognition in clinical and research settings using low-cost sensor data.

Proposed method

• Employing a ceiling-mounted RGB camera to capture overhead video of participants performing the BDT, with a green table surface and larger blocks to improve detection accuracy.
• Using the TensorFlow Object Detection API with a retrained SSD MobileNet model to detect hands and filter out frames with hand occlusions (IoU threshold > 0.3).
• Applying OpenCV-based geometric transformations to correct perspective distortion and localize the blue outline of the target design for block positioning.
• Dividing the corrected image into n×n sub-images and further subdividing each into quadrants to classify block color and orientation (e.g., NW, NE) based on pixel color distribution.
• Implementing the CABS system to analyze block sequence data, detecting errors via label changes at fixed positions and computing metrics like reaction time, spatial distance, and progression patterns.
• Generating visualizations such as scatter plots correlating error counts with completion time to support clinical interpretation and comparison across individuals.

Experimental results

Research questions

• RQ1 Can automated computer vision techniques accurately detect and classify block placements and movements in real time during the Block Design Test using overhead RGB video?
• RQ2 To what extent can AI-based analysis quantify behavioral features such as errors, reaction times, and problem-solving strategies in BDT performance?
• RQ3 How do quantitative metrics derived from block sequences (e.g., error frequency, movement patterns) correlate with completion time and performance quality?
• RQ4 Can the proposed framework support clinical neuropsychologists by providing objective, visualizable summaries of cognitive behavior beyond accuracy and time?

Key findings

• The ABIS system successfully achieved frame-level detection of block positions and orientations, even under challenging conditions such as partial occlusion and perspective distortion.
• The CABS system detected errors by identifying changes in block labels at fixed positions over time, such as swapping a NW block for a NE block, enabling automated error counting.
• Participant A completed puzzles significantly faster and with fewer errors than participant B, with CABS visualizations clearly distinguishing performance outliers like participant C, who made over 20 errors on one puzzle.
• A scatter plot of error count versus completion time revealed distinct performance patterns across participants, with one participant (C) showing a major outlier in both error frequency and duration.
• The framework demonstrated feasibility in a proof-of-concept study with seven college student participants, using annotated video data and simulated strategies for comparison.
• The integration of additional sensor modalities—such as depth data and eye-tracking—was identified as a key future enhancement to improve detection robustness and cognitive insight.