[Paper Review] Rendering of Eyes for Eye-Shape Registration and Gaze Estimation

Erroll Wood, Tadas Baltrušaitis+4 more|arXiv (Cornell University)|Feb 17, 2015

Gaze Tracking and Assistive TechnologyComputer Science21 references64 citations

TL;DR

This paper presents SynthesEyes, a learning-by-synthesis framework that generates photorealistic, fully annotated eye-region images using dynamic, controllable 3D eye models derived from high-resolution head scans. By simulating diverse head poses, gaze directions, and realistic illumination, the method produces high-quality training data that outperforms state-of-the-art approaches in eye-shape registration and cross-dataset gaze estimation, demonstrating the effectiveness of synthetic data with precise control over appearance and geometry.

ABSTRACT

Images of the eye are key in several computer vision problems, such as shape registration and gaze estimation. Recent large-scale supervised methods for these problems require time-consuming data collection and manual annotation, which can be unreliable. We propose synthesizing perfectly labelled photo-realistic training data in a fraction of the time. We used computer graphics techniques to build a collection of dynamic eye-region models from head scan geometry. These were randomly posed to synthesize close-up eye images for a wide range of head poses, gaze directions, and illumination conditions. We used our model's controllability to verify the importance of realistic illumination and shape variations in eye-region training data. Finally, we demonstrate the benefits of our synthesized training data (SynthesEyes) by out-performing state-of-the-art methods for eye-shape registration as well as cross-dataset appearance-based gaze estimation in the wild.

Motivation & Objective

• Address the challenge of collecting and annotating large-scale, high-quality eye image datasets for computer vision tasks.
• Overcome limitations of existing synthetic data, which often lack realistic illumination, material properties, and geometric detail.
• Enable training of robust models for eye-shape registration and appearance-based gaze estimation using fully controllable, photorealistic synthetic data.
• Demonstrate that synthetic data can achieve performance comparable or superior to real-world datasets in cross-dataset evaluation settings.

Proposed method

• Constructed dynamic, high-fidelity 3D eye-region models from dense 3D head scans (1.4M polygons), retopologized to 9,005 polygons for animation.
• Restored fine skin surface details using high-resolution displacement maps and manually annotated 3D landmarks for iris and eyelids.
• Employed image-based lighting (IBL) to simulate realistic illumination variations across diverse environments and lighting conditions.
• Randomly sampled head poses, gaze directions, and lighting configurations to generate a large-scale, diverse training dataset (SynthesEyes).
• Used the controllable nature of the model to verify the impact of realistic illumination and shape variation on model performance.
• Trained two separate models on SynthesEyes: a deformable eye-region model for shape registration and a CNN for appearance-based gaze estimation.

Experimental results

Research questions

• RQ1 Can synthetic, photorealistic eye images with precise ground-truth annotations improve performance in eye-shape registration compared to real-world data?
• RQ2 To what extent does realistic illumination variation in synthetic training data enhance the generalization of gaze estimation models?
• RQ3 How does training on synthetically generated data compare to real-world datasets in cross-dataset gaze estimation scenarios?
• RQ4 Can targeted data synthesis—focusing on specific head pose and gaze ranges—improve performance for real-world applications like laptop-based gaze interaction?
• RQ5 How do variations in eye shape and skin color across individuals affect the generalization of appearance-based gaze estimation models trained on synthetic data?

Key findings

• The SynthesEyes dataset achieved a mean error of 13.91° on the MPIIGaze dataset for cross-dataset gaze estimation, matching the performance of models trained on the real UT Multiview dataset (13.55°).
• Targeted synthesis for laptop-based gaze interaction (10° pose, 20° gaze variation) reduced mean error to 7.90°, significantly outperforming the UT dataset (p < 0.0001).
• Fine-tuning a CNN initially trained on SynthesEyes with real UT data further reduced error to 11.12° (untargeted) and 7.90° (targeted), surpassing the state-of-the-art result (13.91°).
• Eye models with mismatched skin tone or eye shape led to poor generalization, with some models (e.g., f3, m2, m4) producing significantly higher errors on MPIIGaze participants.
• The performance gap between within-dataset and cross-dataset training remains, indicating that synthetic data alone may not fully capture real-world variability.
• The study confirms that realistic illumination and geometric variation are critical for robust model generalization, and that synthetic data with high controllability can effectively support complex vision tasks.