[Paper Review] Self-Learning Transformations for Improving Gaze and Head Redirection

Yufeng Zheng, Seonwook Park+3 more|arXiv (Cornell University)|Oct 23, 2020

Face recognition and analysisComputer Science58 references24 citations

TL;DR

This paper proposes a self-transforming encoder-decoder architecture that disentangles and controls multiple facial factors—including gaze direction, head orientation, lighting, and hue—through self-predicted pseudo-conditions in a latent bottleneck. The method achieves state-of-the-art performance in gaze and head redirection fidelity and improves semi-supervised cross-dataset gaze estimation by augmenting real data with redirected images.

ABSTRACT

Many computer vision tasks rely on labeled data. Rapid progress in generative modeling has led to the ability to synthesize photorealistic images. However, controlling specific aspects of the generation process such that the data can be used for supervision of downstream tasks remains challenging. In this paper we propose a novel generative model for images of faces, that is capable of producing high-quality images under fine-grained control over eye gaze and head orientation angles. This requires the disentangling of many appearance related factors including gaze and head orientation but also lighting, hue etc. We propose a novel architecture which learns to discover, disentangle and encode these extraneous variations in a self-learned manner. We further show that explicitly disentangling task-irrelevant factors results in more accurate modelling of gaze and head orientation. A novel evaluation scheme shows that our method improves upon the state-of-the-art in redirection accuracy and disentanglement between gaze direction and head orientation changes. Furthermore, we show that in the presence of limited amounts of real-world training data, our method allows for improvements in the downstream task of semi-supervised cross-dataset gaze estimation. Please check our project page at: https://ait.ethz.ch/projects/2020/STED-gaze/

Motivation & Objective

• To address the challenge of controlling fine-grained facial attributes like gaze and head orientation in in-the-wild images where paired data is unavailable.
• To disentangle task-relevant (gaze, head pose) from task-irrelevant (lighting, hue, etc.) factors in a self-supervised manner.
• To develop a principled evaluation scheme for measuring redirection accuracy and disentanglement fidelity.
• To improve semi-supervised cross-dataset gaze estimation by augmenting limited real-world training data using the proposed redirection framework.

Proposed method

• Proposes a self-transforming encoder-decoder (ST-ED) architecture with multiple transformable latent factors, each consisting of a latent embedding and a self-predicted pseudo-condition.
• Uses self-predicted pseudo-conditions to reduce reliance on noisy or imperfect labels during conditional image translation.
• Applies novel constraints to enforce disentanglement among independent factors while preserving precise control over target gaze and head orientation.
• Introduces a redirection error metric to quantify how accurately target gaze and head orientation are reproduced in generated images.
• Introduces a task disentanglement error metric to measure how much gaze or head orientation changes when extraneous factors are altered.
• Trains the gaze redirection model in a semi-supervised manner using limited real data, then uses it to augment training data for downstream gaze estimation.

Experimental results

Research questions

• RQ1 Can a self-supervised generative model effectively disentangle and control multiple facial factors—including gaze, head orientation, lighting, and hue—without paired supervision?
• RQ2 How does explicit disentanglement of task-irrelevant factors improve the accuracy and faithfulness of gaze and head redirection?
• RQ3 To what extent can redirected images generated by this method improve performance in semi-supervised cross-dataset gaze estimation?
• RQ4 How does the proposed evaluation scheme compare to existing metrics in measuring redirection fidelity and disentanglement?

Key findings

• The proposed method achieves state-of-the-art redirection accuracy, outperforming He et al. [17] and StarGAN [21] in both qualitative and quantitative evaluations on the GazeCapture dataset.
• The method reduces redirection error by 25% compared to the next best baseline, demonstrating superior control over target gaze and head orientation.
• The task disentanglement error metric shows that the model maintains gaze and head orientation stability even when lighting and hue are varied, confirming effective disentanglement.
• In semi-supervised cross-dataset gaze estimation, the method improves mean absolute error (MAE) by up to 15% on four benchmark datasets by augmenting real training data with redirected images.
• The model generalizes well to challenging cases such as large head poses, eyeglasses, and blurry inputs, producing photo-realistic outputs.
• The ablation study confirms that disentangling extraneous factors leads to more accurate and robust redirection, validating the core design principle.