[Paper Review] Adaptive Feature Fusion Network for Gaze Tracking in Mobile Tablets

Yiwei Bao, Yihua Cheng+2 more|arXiv (Cornell University)|Mar 20, 2021

Gaze Tracking and Assistive Technology41 references4 citations

TL;DR

This paper proposes Adaptive Feature Fusion Network (AFF-Net), a novel CNN-based gaze tracking method for mobile tablets that adaptively fuses left and right eye features using stacked feature maps and Squeeze-and-Excitation layers, while employing Adaptive Group Normalization to recalibrate eye features using facial appearance. The method achieves state-of-the-art performance on GazeCapture and MPIIFaceGaze datasets, with 1.62 cm and 2.30 cm Euclidean error on phone and tablet images, respectively.

ABSTRACT

Recently, many multi-stream gaze estimation methods have been proposed. They estimate gaze from eye and face appearances and achieve reasonable accuracy. However, most of the methods simply concatenate the features extracted from eye and face appearance. The feature fusion process has been ignored. In this paper, we propose a novel Adaptive Feature Fusion Network (AFF-Net), which performs gaze tracking task in mobile tablets. We stack two-eye feature maps and utilize Squeeze-and-Excitation layers to adaptively fuse two-eye features according to their similarity on appearance. Meanwhile, we also propose Adaptive Group Normalization to recalibrate eye features with the guidance of facial feature. Extensive experiments on both GazeCapture and MPIIFaceGaze datasets demonstrate consistently superior performance of the proposed method.

Motivation & Objective

• To improve gaze estimation accuracy in mobile tablets by leveraging the structural similarity between left and right eyes.
• To address the limitation of simple concatenation of eye features by introducing a spatial-aware fusion mechanism that preserves feature map structure.
• To enhance eye feature extraction by incorporating facial appearance characteristics through adaptive normalization.
• To develop a unified framework that jointly optimizes eye and face feature utilization for robust gaze estimation in unconstrained environments.

Proposed method

• The method stacks feature maps from left and right eyes at different network layers to preserve spatial relationships and enable more effective fusion.
• Squeeze-and-Excitation (SE) layers are applied after stacking to adaptively recalibrate channel-wise feature importance based on inter-eye appearance similarity.
• A novel Adaptive Group Normalization (AdaGN) layer is introduced, which uses face and eye bounding boxes and facial features to dynamically compute scale and shift parameters for eye feature recalibration.
• The network processes both eye and face inputs end-to-end, with the face stream providing contextual guidance to refine eye feature representations.
• The final gaze prediction is generated from the fused eye features, with the model trained to minimize 2D gaze position error on real-world mobile tablet images.
• The architecture is evaluated on two large-scale datasets: GazeCapture (mobile phone and tablet) and MPIIFaceGaze (laptop-based).

Experimental results

Research questions

• RQ1 Can stacking eye feature maps instead of concatenating feature vectors improve gaze estimation accuracy by preserving spatial structure?
• RQ2 Does adaptive channel-wise weighting via Squeeze-and-Excitation layers enhance performance by modeling inter-eye appearance similarity?
• RQ3 Can facial appearance characteristics (e.g., head pose, lighting, identity) be effectively used to guide eye feature learning through adaptive normalization?
• RQ4 How does the proposed fusion and guidance mechanism perform under extreme conditions, such as small face sizes or varying gaze directions?

Key findings

• AFF-Net achieves a 1.62 cm Euclidean error on phone images and 2.30 cm on tablet images from the GazeCapture dataset, outperforming state-of-the-art methods.
• On the MPIIFaceGaze dataset, the method achieves 3.9 cm Euclidean error and 4.4 degree angular error in 3D gaze direction estimation, demonstrating strong generalization to laptop-based settings.
• Ablation studies show that removing any component—stacking, SE layers, or AdaGN—increases error by approximately 4%, confirming the contribution of each module.
• Heatmap analysis reveals that AFF-Net maintains lower error across all gaze locations, especially near the camera edge, indicating improved robustness.
• The model shows superior performance on small face sizes, with significantly lower error compared to baselines when face width is minimal, indicating strong generalization to extreme cases.