[Paper Review] MagicEyes: A Large Scale Eye Gaze Estimation Dataset for Mixed Reality

Zhengyang Wu, Srivignesh Rajendran+4 more|arXiv (Cornell University)|Mar 18, 2020

Gaze Tracking and Assistive Technology37 references4 citations

TL;DR

This paper introduces MagicEyes, a large-scale eye gaze estimation dataset collected using real mixed reality devices, featuring 587 subjects, 80,000 human-labeled images, and over 800,000 gaze target labels. It proposes EyeNet, a multi-task deep learning model that jointly estimates cornea center, glints, pupil, and eye segmentation in a single forward pass, demonstrating improved robustness and efficiency over traditional heuristic-based pipelines on off-axis eye images.

ABSTRACT

With the emergence of Virtual and Mixed Reality (XR) devices, eye tracking has received significant attention in the computer vision community. Eye gaze estimation is a crucial component in XR -- enabling energy efficient rendering, multi-focal displays, and effective interaction with content. In head-mounted XR devices, the eyes are imaged off-axis to avoid blocking the field of view. This leads to increased challenges in inferring eye related quantities and simultaneously provides an opportunity to develop accurate and robust learning based approaches. To this end, we present MagicEyes, the first large scale eye dataset collected using real MR devices with comprehensive ground truth labeling. MagicEyes includes $587$ subjects with $80,000$ images of human-labeled ground truth and over $800,000$ images with gaze target labels. We evaluate several state-of-the-art methods on MagicEyes and also propose a new multi-task EyeNet model designed for detecting the cornea, glints and pupil along with eye segmentation in a single forward pass.

Motivation & Objective

• To address the lack of large-scale, real-world datasets for off-axis eye gaze estimation in mixed reality (MR) devices.
• To benchmark state-of-the-art methods on a comprehensive, real-device collected dataset with rich annotations.
• To develop a unified deep learning framework that jointly estimates key eye features (pupil, glints, cornea, segmentation) for improved robustness and efficiency.
• To enable end-to-end learning-based gaze estimation that outperforms hybrid geometric and heuristic-based approaches in real-world settings.

Proposed method

• The MagicEyes dataset was collected using real MR devices, capturing 640×480 IR eye images from 587 subjects under diverse lighting and demographic conditions.
• The dataset includes 80,000 images with human-verified ground truth for eye part segmentation (face, sclera, iris, pupil) and over 800,000 images with gaze target labels.
• EyeNet is a multi-task deep neural network with a shared ResNet-50 + FPN encoder and four task-specific decoders for eye segmentation, pupil and glint localization, and cornea center estimation.
• The model uses a single forward pass to predict multiple outputs, leveraging shared features to improve generalization and reduce computational cost.
• The network integrates supervised learning with geometric constraints, using a differentiable 3D eye model to refine gaze estimation from 2D predictions.
• Evaluation includes ablation studies and comparisons with SOTA methods like RlTNet and NVGaze on segmentation, pupil localization, and gaze estimation tasks.

Experimental results

Research questions

• RQ1 How do state-of-the-art eye gaze estimation methods perform on a large-scale, real-world dataset collected from actual mixed reality devices?
• RQ2 Can a multi-task deep learning model jointly estimate eye features (pupil, glints, cornea center, segmentation) more robustly than separate or heuristic-based pipelines?
• RQ3 To what extent does end-to-end learning with geometric priors improve gaze estimation accuracy in off-axis, real-world eye imaging conditions?
• RQ4 What are the limitations of current end-to-end deep learning approaches for gaze estimation compared to hybrid geometric methods?

Key findings

• MagicEyes is the largest publicly available eye gaze estimation dataset collected with real MR devices, featuring 587 subjects and over 800,000 gaze target labels.
• Fully convolutional networks achieved strong performance on eye segmentation and pupil localization, indicating promise for geometric methods in real-world settings.
• Direct end-to-end deep learning methods like NVGaze showed high variance and were unsuitable for real-world deployment, despite strong performance in ideal conditions.
• The proposed EyeNet model outperformed baseline methods by jointly estimating cornea center, glints, pupil, and segmentation, reducing reliance on hand-crafted heuristics.
• The integration of geometric constraints into the learning process improved the robustness and accuracy of gaze estimation, especially under challenging off-axis viewing conditions.
• The study demonstrates that multi-task learning with shared representations enables more efficient and accurate estimation of multiple eye-related quantities in a single inference pass.