[Paper Review] Appearance-based Gaze Estimation With Deep Learning: A Review and Benchmark

Yihua Cheng, Haofei Wang+2 more|arXiv (Cornell University)|Apr 26, 2021

Gaze Tracking and Assistive TechnologyComputer Science122 references67 citations

TL;DR

A comprehensive review and benchmark of appearance-based gaze estimation methods using deep learning, covering feature extraction, network design, calibration, and devices, with a standardized evaluation pipeline.

ABSTRACT

Human gaze provides valuable information on human focus and intentions, making it a crucial area of research. Recently, deep learning has revolutionized appearance-based gaze estimation. However, due to the unique features of gaze estimation research, such as the unfair comparison between 2D gaze positions and 3D gaze vectors and the different pre-processing and post-processing methods, there is a lack of a definitive guideline for developing deep learning-based gaze estimation algorithms. In this paper, we present a systematic review of the appearance-based gaze estimation methods using deep learning. Firstly, we survey the existing gaze estimation algorithms along the typical gaze estimation pipeline: deep feature extraction, deep learning model design, personal calibration and platforms. Secondly, to fairly compare the performance of different approaches, we summarize the data pre-processing and post-processing methods, including face/eye detection, data rectification, 2D/3D gaze conversion and gaze origin conversion. Finally, we set up a comprehensive benchmark for deep learning-based gaze estimation. We characterize all the public datasets and provide the source code of typical gaze estimation algorithms. This paper serves not only as a reference to develop deep learning-based gaze estimation methods, but also a guideline for future gaze estimation research. The project web page can be found at https://phi-ai.buaa.edu.cn/Gazehub.

Motivation & Objective

• Survey deep learning-based appearance-based gaze estimation methods across input types (eye, face, video).
• Analyze network architectures and supervision schemes (supervised, semi/self/unsupervised, multi-task, recurrent).
• Evaluate data pre-processing and post-processing steps and their impact on performance.
• Provide a unified gaze estimation benchmark with publicly available code and datasets.
• Offer guidelines and future research directions for robust, cross-subject gaze estimation.

Proposed method

• Classify input features into eye images, face images, and videos for feature extraction review.
• Discuss CNN architectures and supervision strategies (supervised, semi/self-/unsupervised, multi-task, recurrent).
• Describe calibration methods and device/platform considerations for gaze estimation.
• Summarize data pre-processing and post-processing steps and gaze conversions used in benchmarks.
• Build and implement a benchmark by collecting public datasets and reproducing typical gaze estimation codes for fair comparison.

Experimental results

Research questions

• RQ1 What are the main deep learning approaches for appearance-based gaze estimation and how do they differ in input types and architectures?
• RQ2 How do calibration, subject variability, and device/platform choices affect gaze estimation performance?
• RQ3 What preprocessing and post-processing steps are essential to fairly compare gaze estimation methods across datasets?
• RQ4 What are the current public datasets and baseline methods, and how do they perform under a unified evaluation protocol?

Key findings

• Deep learning methods improve robustness to head motion and cross-subject variation compared to conventional appearance-based methods.
• Using two-eye inputs with attention-based fusion or asymmetric regression can improve gaze estimation accuracy.
• Calibration, data augmentation, and subject-invariant feature learning are critical for cross-subject performance.
• A benchmark is built by collecting public datasets and re-implementing typical gaze estimation algorithms to enable fair comparison.
• The paper provides implemented methods and data processing code at phi-ai.org/GazeHub for reproducibility.