[Paper Review] A Generalized and Robust Method Towards Practical Gaze Estimation on Smart Phone

Tianchu Guo, Yongchao Liu+6 more|arXiv (Cornell University)|Oct 16, 2019

Gaze Tracking and Assistive Technology31 references4 citations

TL;DR

This paper proposes a Tolerant and Talented (TAT) training scheme and a Disturbance with Ordinal loss (DwO) method to address overfitting and jittery gaze estimation on smartphones. TAT uses iterative knowledge distillation with cosine similarity pruning and aligned orthogonal initialization to reduce overfitting and improve generalization, while DwO enhances robustness via adversarial training with ordinal loss. The methods achieve state-of-the-art accuracy on GazeCapture and significantly reduce prediction jitter on a new robustness benchmark.

ABSTRACT

Gaze estimation for ordinary smart phone, e.g. estimating where the user is looking at on the phone screen, can be applied in various applications. However, the widely used appearance-based CNN methods still have two issues for practical adoption. First, due to the limited dataset, gaze estimation is very likely to suffer from over-fitting, leading to poor accuracy at run time. Second, the current methods are usually not robust, i.e. their prediction results having notable jitters even when the user is performing gaze fixation, which degrades user experience greatly. For the first issue, we propose a new tolerant and talented (TAT) training scheme, which is an iterative random knowledge distillation framework enhanced with cosine similarity pruning and aligned orthogonal initialization. The knowledge distillation is a tolerant teaching process providing diverse and informative supervision. The enhanced pruning and initialization is a talented learning process prompting the network to escape from the local minima and re-born from a better start. For the second issue, we define a new metric to measure the robustness of gaze estimator, and propose an adversarial training based Disturbance with Ordinal loss (DwO) method to improve it. The experimental results show that our TAT method achieves state-of-the-art performance on GazeCapture dataset, and that our DwO method improves the robustness while keeping comparable accuracy.

Motivation & Objective

• Address the performance gap between training and testing due to overfitting in smartphone gaze estimation.
• Reduce prediction jitter during gaze fixation, which degrades user experience in real-time applications.
• Develop a new metric and dataset to quantitatively evaluate gaze estimator robustness.
• Improve generalization and robustness through novel training schemes without sacrificing accuracy.
• Enable practical deployment of gaze estimation on consumer smartphones.

Proposed method

• Propose an iterative random knowledge distillation framework as a tolerant teaching process, using diverse and informative supervision from randomly selected teacher networks.
• Introduce cosine similarity pruning to remove redundant, low-utility weights, enhancing model efficiency and generalization.
• Develop aligned orthogonal initialization (AOI) to reinitialize weights orthogonal to each other and aligned with the original distribution, enabling better optimization restarts.
• Design a new robustness metric (MSD) and collect a new dataset (GazeStare) to evaluate jitter and robustness.
• Propose Disturbance with Ordinal loss (DwO), a customized adversarial training method that applies perturbations to ordinal loss bins to improve robustness.
• Use PGD-based adversarial training with controlled perturbation strength and data retention (Org.%), tuned via hyperparameter search.

Experimental results

Research questions

• RQ1 Can an iterative knowledge distillation framework with pruning and reinitialization reduce overfitting in gaze estimation on limited datasets?
• RQ2 Does cosine similarity pruning and aligned orthogonal initialization improve model generalization and convergence in gaze estimation?
• RQ3 Can a new metric and dataset effectively quantify and evaluate gaze prediction jitter in real-world conditions?
• RQ4 Does adversarial training with ordinal loss (DwO) significantly reduce jitter while maintaining high accuracy?
• RQ5 Can the proposed TAT and DwO methods achieve state-of-the-art performance on benchmark datasets like GazeCapture and GazeStare?

Key findings

• The TAT training scheme achieves state-of-the-art performance on the GazeCapture dataset, outperforming existing methods.
• The DwO method reduces mean squared deviation (MSD) by 19.2% compared to the baseline while maintaining comparable accuracy (1.20 cm vs. 1.15 cm error).
• Using T=1 and Org.%=90% in DwO achieves the best trade-off between robustness and accuracy, reducing MSD from 0.42 to 0.36 cm.
• The proposed aligned orthogonal initialization method results in more stable weight distributions, with L2 norms closely matching those of pruned weights, preventing network collapse.
• The model trained with DwO shows significantly less jitter in prediction points during fixation, as visualized in Fig. 1(b), with predictions more concentrated around the true gaze point.
• The number of center bins in ordinal loss is critical: 8 bins yield optimal performance, balancing perturbation effectiveness and model stability.