[Paper Review] LNSMM: Eye Gaze Estimation With Local Network Share Multiview Multitask

Yong Huang, Ben M. Chen+1 more|arXiv (Cornell University)|Jan 18, 2021

Gaze Tracking and Assistive Technology20 references4 citations

TL;DR

This paper proposes LNSMM, a novel deep learning framework for simultaneous eye gaze point and gaze direction estimation using a local shared network and multiview multitask learning. By leveraging coplanar constraints for gaze direction and cross-view pooling for gaze point estimation, the method achieves state-of-the-art performance on both gaze point (3.9° average error) and gaze direction (4.70°) tasks, outperforming existing methods on custom and public datasets.

ABSTRACT

Eye gaze estimation has become increasingly significant in computer vision.In this paper,we systematically study the mainstream of eye gaze estimation methods,propose a novel methodology to estimate eye gaze points and eye gaze directions simultaneously.First,we construct a local sharing network for feature extraction of gaze points and gaze directions estimation,which can reduce network computational parameters and converge quickly;Second,we propose a Multiview Multitask Learning (MTL) framework,for gaze directions,a coplanar constraint is proposed for the left and right eyes,for gaze points,three views data input indirectly introduces eye position information,a cross-view pooling module is designed, propose joint loss which handle both gaze points and gaze directions estimation.Eventually,we collect a dataset to use of gaze points,which have three views to exist public dataset.The experiment show our method is state-of-the-art the current mainstream methods on two indicators of gaze points and gaze directions.

Motivation & Objective

• To address the limitation of existing gaze estimation methods that treat gaze point and gaze direction estimation separately, ignoring their interdependence.
• To improve accuracy and robustness in eye gaze estimation by jointly modeling gaze point and gaze direction using multiview input.
• To reduce model complexity and accelerate convergence through a local shared network architecture.
• To enhance performance under challenging conditions such as occlusion and specular reflections using multiview data.
• To validate the method on a newly collected multiview dataset with three views per subject, including both with and without glasses.

Proposed method

• A local shared network is designed to extract features for both gaze point and gaze direction estimation, reducing parameters and improving training efficiency.
• A multiview multitask learning (MTL) framework is proposed, where left and right eye gaze directions are constrained to lie on a common plane.
• For gaze point estimation, a cross-view pooling module fuses features from three views to implicitly encode eye position information.
• A joint loss function is introduced to optimize both gaze point and gaze direction estimation simultaneously.
• The method uses multi-view input (three perspectives) to improve robustness to occlusion and specular reflections, especially in glasses-wearing scenarios.
• The backbone network is based on ResNeXt-36 with local weight sharing, enhancing feature representation while reducing computational cost.

Experimental results

Research questions

• RQ1 How can gaze point and gaze direction estimation be jointly optimized to improve accuracy and exploit their interdependence?
• RQ2 What is the impact of multiview input on reducing noise and occlusion in eye gaze estimation?
• RQ3 Can a local shared network architecture effectively reduce model parameters and accelerate convergence without sacrificing performance?
• RQ4 How does the coplanar constraint for left and right eye gaze directions improve estimation accuracy?
• RQ5 To what extent does the proposed method generalize across different eye conditions, such as wearing glasses?

Key findings

• The proposed LNSMM method achieves a 3.9° average error in gaze point estimation on the authors' custom multiview dataset, outperforming all compared methods.
• On the MPIIGaze dataset, the method achieves a 4.70° average angle error for gaze direction estimation, surpassing the next best method by 0.61°.
• The use of multiview input improves performance, especially for subjects wearing glasses, where occlusion and reflections are more prominent.
• The local shared network (ResNeXt-36) reduces model complexity and improves convergence, with the method showing superior performance compared to non-shared variants.
• The cross-view pooling module effectively integrates multi-view information, enhancing gaze point estimation accuracy by leveraging eye position cues.
• The coplanar constraint for left and right eyes significantly improves gaze direction estimation, particularly in symmetric and consistent 3D gaze modeling.