[Paper Review] A Deep Learning Approach for the Segmentation of Electroencephalography Data in Eye Tracking Applications

Lukas Wolf, Ard Kastrati+6 more|arXiv (Cornell University)|Jun 17, 2022

Gaze Tracking and Assistive Technology4 citations

TL;DR

This paper introduces DETRtime, a novel deep learning framework that performs end-to-end segmentation of EEG data into ocular events—fixations, saccades, and blinks—without requiring video-based eye-tracking data. Leveraging a time-series adaptation of the DETR object detection architecture, it achieves state-of-the-art performance across four eye-tracking paradigms and generalizes effectively to EEG sleep stage segmentation.

ABSTRACT

The collection of eye gaze information provides a window into many critical aspects of human cognition, health and behaviour. Additionally, many neuroscientific studies complement the behavioural information gained from eye tracking with the high temporal resolution and neurophysiological markers provided by electroencephalography (EEG). One of the essential eye-tracking software processing steps is the segmentation of the continuous data stream into events relevant to eye-tracking applications, such as saccades, fixations, and blinks. Here, we introduce DETRtime, a novel framework for time-series segmentation that creates ocular event detectors that do not require additionally recorded eye-tracking modality and rely solely on EEG data. Our end-to-end deep learning-based framework brings recent advances in Computer Vision to the forefront of the times series segmentation of EEG data. DETRtime achieves state-of-the-art performance in ocular event detection across diverse eye-tracking experiment paradigms. In addition to that, we provide evidence that our model generalizes well in the task of EEG sleep stage segmentation.

Motivation & Objective

• To develop a deep learning framework that segments EEG data into ocular events (fixations, saccades, blinks) without relying on video-based eye-tracking data.
• To adapt the DETR architecture—originally designed for object detection in images—into a time-series segmentation model for EEG signals.
• To create a unified, end-to-end method capable of detecting multiple ocular events with high temporal resolution (2 ms) and precise onset/offset localization.
• To evaluate the model’s performance across diverse experimental paradigms and demonstrate its generalization to other time-series segmentation tasks, such as sleep staging.
• To provide a publicly available benchmark dataset with 139k fixations, 139k saccades, and 14k blinks, paired with EEG and infrared eye-tracking ground truth.

Proposed method

• DETRtime is a transformer-based architecture adapted for sequential EEG data, mapping variable-length time-series inputs to sequences of class labels.
• The model uses a learnable object query mechanism and a set prediction head to predict multiple ocular events simultaneously, similar to instance segmentation in computer vision.
• It employs a multi-head self-attention mechanism to capture long-range dependencies in EEG signals, enabling precise localization of event boundaries.
• The framework is trained end-to-end using a Hungarian loss for matching predictions to ground-truth event instances, ensuring accurate detection of onset and offset times.
• The model is evaluated with a 2 ms temporal resolution, allowing fine-grained segmentation of ocular events.
• Multiple backbones (e.g., EEGNet, EEG-TCN) are evaluated to identify optimal feature extraction components for EEG data.

Experimental results

Research questions

• RQ1 Can a transformer-based architecture like DETR be effectively adapted for time-series segmentation of EEG data to detect ocular events without external eye-tracking data?
• RQ2 How does DETRtime compare to existing state-of-the-art models in terms of precision, recall, and F1 score for ocular event detection across diverse experimental paradigms?
• RQ3 To what extent does DETRtime generalize to other time-series segmentation tasks, such as EEG sleep stage classification?
• RQ4 What are the key challenges in EEG-based ocular event detection, particularly in distinguishing between similar events like vertical saccades and blinks?
• RQ5 Can a unified, end-to-end deep learning framework eliminate the need for separate preprocessing or hand-crafted feature engineering pipelines in EEG-based ocular event segmentation?

Key findings

• DETRtime achieved an average precision of 0.936 and recall of 0.902 across four eye-tracking paradigms, significantly outperforming existing state-of-the-art models.
• The model demonstrated strong generalization by outperforming current SOTA methods on the Sleep-EDF-153 dataset for EEG sleep stage segmentation.
• DETRtime achieved higher F1 scores for fixations compared to saccades and blinks, reflecting the challenge of distinguishing between similar signal patterns, especially vertical saccades and blinks.
• The ablation studies confirmed that a separate segmentation pipeline is unnecessary when using DETRtime, as the end-to-end architecture inherently handles event localization.
• The model’s performance was robust across diverse EEG signal qualities and participant variability, though fine-tuning per participant may further improve results.
• The proposed benchmark dataset with 168 participants and 292k labeled ocular events provides a valuable resource for future research in EEG-based eye movement analysis.