[Paper Review] OpenEDS2020: Open Eyes Dataset

Cristina Palmero, Abhishek Sharma+4 more|arXiv (Cornell University)|May 8, 2020

Gaze Tracking and Assistive Technology25 references5 citations

TL;DR

OpenEDS2020 is a large-scale, anonymized dataset of high-resolution (640×400), 100 Hz eye-image sequences captured via dual synchronized eye-tracking cameras in a VR environment. It enables research in spatio-temporal gaze prediction and eye segmentation, achieving a 5.37° average angular error in gaze prediction and 84.1% mIoU in semantic segmentation, serving as a benchmark for real-time VR applications.

ABSTRACT

We present the second edition of OpenEDS dataset, OpenEDS2020, a novel dataset of eye-image sequences captured at a frame rate of 100 Hz under controlled illumination, using a virtual-reality head-mounted display mounted with two synchronized eye-facing cameras. The dataset, which is anonymized to remove any personally identifiable information on participants, consists of 80 participants of varied appearance performing several gaze-elicited tasks, and is divided in two subsets: 1) Gaze Prediction Dataset, with up to 66,560 sequences containing 550,400 eye-images and respective gaze vectors, created to foster research in spatio-temporal gaze estimation and prediction approaches; and 2) Eye Segmentation Dataset, consisting of 200 sequences sampled at 5 Hz, with up to 29,500 images, of which 5% contain a semantic segmentation label, devised to encourage the use of temporal information to propagate labels to contiguous frames. Baseline experiments have been evaluated on OpenEDS2020, one for each task, with average angular error of 5.37 degrees when performing gaze prediction on 1 to 5 frames into the future, and a mean intersection over union score of 84.1% for semantic segmentation. As its predecessor, OpenEDS dataset, we anticipate that this new dataset will continue creating opportunities to researchers in eye tracking, machine learning and computer vision communities, to advance the state of the art for virtual reality applications. The dataset is available for download upon request at http://research.fb.com/programs/openeds-2020-challenge/.

Motivation & Objective

• To address the lack of high-resolution, temporally dense, and anonymized eye-tracking datasets suitable for real-time VR/AR applications.
• To enable research in spatio-temporal gaze estimation and prediction, particularly for compensating delays in foveated rendering.
• To support development of robust eye segmentation models using temporal consistency and sparse annotations.
• To provide a benchmark dataset that advances the state of the art in computer vision, machine learning, and eye tracking for immersive technologies.
• To foster reproducible research through a publicly available, privacy-preserving dataset with controlled illumination and diverse participant demographics.

Proposed method

• The dataset was collected using a VR HMD equipped with two synchronized eye-facing cameras at 100 Hz, capturing 80 participants performing gaze-elicited tasks.
• The Gaze Prediction Dataset includes up to 66,560 sequences of 550,400 eye images with corresponding 3D gaze vectors for temporal modeling.
• The Eye Segmentation Dataset contains 200 sequences sampled at 5 Hz, with 5% of frames having dense semantic segmentation masks for sclera, iris, pupil, and background.
• A 3D gaze prediction baseline was implemented using a CNN with a 32-unit FC layer and a 2-unit linear regression head, trained on 75% of data with data weighting to handle class imbalance.
• Gaze prediction used a 50-frame sliding window to compute linear regression parameters for predicting the next 5 frames into the future.
• A semantic segmentation baseline used a lightweight encoder-decoder CNN with depthwise separable convolutions and multiplicative skip connections, trained on 1,605 annotated images.

Experimental results

Research questions

• RQ1 Can a simple linear regression model effectively predict future gaze positions using only 50 frames of past gaze estimates?
• RQ2 How does the performance of gaze prediction degrade over time, particularly during fast eye movements like saccades?
• RQ3 To what extent can temporal information improve semantic segmentation accuracy in eye images when only 5% of frames are fully annotated?
• RQ4 Can a lightweight deep learning model achieve high segmentation accuracy (mIoU) on sparse annotations while maintaining low computational cost?
• RQ5 How does the quality of eye segmentation and gaze estimation correlate with the performance of downstream VR applications such as foveated rendering?

Key findings

• The gaze prediction baseline achieved an average angular error of 5.37 degrees when predicting gaze 50ms into the future, with error increasing gradually over time.
• The p95 error for gaze prediction reached 12.48 degrees, indicating significant challenges in predicting fast saccadic movements with linear models.
• The semantic segmentation baseline achieved a mean intersection over union (mIoU) score of 84.1% on the test set when using only sparse annotations and no temporal modeling.
• The model achieved 97.1% mIoU for background, 67.4% for sclera, 88.2% for iris, and 83.5% for pupil, indicating strong performance on most structures.
• The trained gaze estimation network achieved a 4.58° average error on the validation set, aligning with state-of-the-art subject-independent methods.
• The dataset's high temporal resolution (100 Hz) and controlled environment enable accurate modeling of fast eye movements such as saccades, which are critical for real-time VR systems.