[Paper Review] A Computer Vision System for Attention Mapping in SLAM based 3D Models

Lucas Paletta, Katrin Santner+5 more|arXiv (Cornell University)|May 6, 2013

Robotics and Sensor-Based Localization20 references6 citations

TL;DR

This paper presents a real-time computer vision system that enables 3D gaze tracking and attention mapping within SLAM-generated 3D environments using RGB-D sensors. By combining RGB-D SLAM with descriptor matching, the system achieves full 3D recovery of gaze direction, view frustum, and region-of-interest (ROI) annotation directly in an automatically reconstructed 3D model, enabling automated human factors analysis in real-world settings.

ABSTRACT

The study of human factors in the frame of interaction studies has been relevant for usability engi-neering and ergonomics for decades. Today, with the advent of wearable eye-tracking and Google glasses, monitoring of human factors will soon become ubiquitous. This work describes a computer vision system that enables pervasive mapping and monitoring of human attention. The key contribu-tion is that our methodology enables full 3D recovery of the gaze pointer, human view frustum and associated human centred measurements directly into an automatically computed 3D model in real-time. We apply RGB-D SLAM and descriptor matching methodologies for the 3D modelling, locali-zation and fully automated annotation of ROIs (regions of interest) within the acquired 3D model. This innovative methodology will open new avenues for attention studies in real world environments, bringing new potential into automated processing for human factors technologies.

Motivation & Objective

• To enable real-time, 3D-aware attention mapping in natural environments using wearable eye-tracking and RGB-D sensors.
• To address the lack of integrated 3D gaze tracking and spatial annotation in real-world environments for human factors studies.
• To automate the detection and labeling of regions of interest (ROIs) within 3D models using computer vision techniques.
• To integrate gaze data with SLAM-based 3D reconstructions for precise, human-centered spatial measurements.

Proposed method

• The system uses RGB-D SLAM to generate a dense, real-time 3D reconstruction of the environment.
• Gaze direction is estimated from monocular eye-tracking data, projected into 3D space using calibrated eye-camera geometry.
• Descriptor matching (e.g., SIFT or similar) is applied to match visual features between eye-tracked scenes and the 3D model for localization.
• Regions of interest (ROIs) are automatically detected and annotated in the 3D model based on gaze density and fixation patterns.
• The system computes the human view frustum in 3D space, enabling spatial analysis of visual attention.
• All processing is performed in real time, enabling continuous monitoring of attention in dynamic, real-world settings.

Experimental results

Research questions

• RQ1 How can gaze data be accurately mapped into a 3D environment reconstructed via SLAM for spatial attention analysis?
• RQ2 What techniques enable real-time, fully automated ROI annotation in 3D models using eye-tracking and visual features?
• RQ3 Can gaze direction and view frustum be recovered in 3D space from wearable eye-tracking and RGB-D data?
• RQ4 How can attention patterns be quantitatively measured and visualized within a 3D scene for human factors evaluation?
• RQ5 What is the feasibility of integrating gaze tracking with SLAM for automated, real-time attention mapping in unstructured environments?

Key findings

• The system successfully achieves real-time 3D recovery of gaze direction and view frustum within SLAM-generated 3D models.
• Automatic ROI detection and annotation are enabled through descriptor matching and gaze pattern clustering.
• The method supports fully automated, human-centered spatial measurements directly in the 3D environment.
• The integration of eye-tracking with RGB-D SLAM allows for persistent, context-aware attention mapping in real-world settings.
• The approach enables new applications in automated human factors analysis, particularly in usability engineering and ergonomics.
• The system demonstrates feasibility for pervasive attention monitoring in natural, dynamic environments.