[Paper Review] Learning a High Fidelity Pose Invariant Model for High-resolution Face Frontalization

Jie Cao, Yibo Hu+3 more|arXiv (Cornell University)|Jun 22, 2018

Face recognition and analysis55 citations

TL;DR

HF-PIM introduces a dense correspondence field and texture map warping to frontalize high-resolution faces, supervised by Adversarial Residual Dictionary Learning to avoid 3D data dependence, achieving state-of-the-art pose-invariant recognition and high-fidelity frontalization.

ABSTRACT

Face frontalization refers to the process of synthesizing the frontal view of a face from a given profile. Due to self-occlusion and appearance distortion in the wild, it is extremely challenging to recover faithful results and preserve texture details in a high-resolution. This paper proposes a High Fidelity Pose Invariant Model (HF-PIM) to produce photographic and identity-preserving results. HF-PIM frontalizes the profiles through a novel texture warping procedure and leverages a dense correspondence field to bind the 2D and 3D surface spaces. We decompose the prerequisite of warping into dense correspondence field estimation and facial texture map recovering, which are both well addressed by deep networks. Different from those reconstruction methods relying on 3D data, we also propose Adversarial Residual Dictionary Learning (ARDL) to supervise facial texture map recovering with only monocular images. Exhaustive experiments on both controlled and uncontrolled environments demonstrate that the proposed method not only boosts the performance of pose-invariant face recognition but also dramatically improves high-resolution frontalization appearances.

Motivation & Objective

• Address the challenge of high-resolution, identity-preserving face frontalization in-the-wild.
• Leverage a dense 2D-3D binding via a dense correspondence field and facial texture maps to guide warping.
• Supervise texture map recovery without 3D data through Adversarial Residual Dictionary Learning (ARDL).
• Provide an end-to-end deep network that unifies geometry estimation, texture recovery, and frontalization.
• Demonstrate improved pose-invariant recognition and high-resolution frontalization on multiple datasets.

Proposed method

• Estimate a dense correspondence field F linking 2D texture space to the frontal view.
• Recover a facial texture map T in UV space via an autoencoder, guided by ARDL when ground-truth T is unavailable.
• Warp the texture map to the frontal view using Y = warp(i,j;F,T) = T_{u_{ij},v_{ij}} and use aRec to generate non-facial regions.
• Train with reconstruction loss L_rec = ||Rec(F,T) - Y||_1 to ensure visual realism.
• Use ARDL to supervise texture representation without 3D data by aligning texture encodings across poses via a dictionary learning module and an adversarial loss against a pose-discriminating dictionary predictor.
• Employ perceptual loss L_p to preserve identity features and an RGB adversarial loss L_g/L_d for photorealism.

Experimental results

Research questions

• RQ1 Can a dense 2D-3D binding via a dense correspondence field improve high-resolution face frontalization from profile images?
• RQ2 Does ARDL enable effective texture map recovery without ground-truth 3D texture maps?
• RQ3 Can the proposed HF-PIM achieve high-fidelity, identity-preserving frontalization at 256×256 resolution in-the-wild?
• RQ4 Does the method improve pose-invariant face recognition on standard benchmarks and high-resolution datasets?

Key findings

• HF-PIM substantially improves rank-1 recognition across large pose variations on Multi-PIE compared to prior methods.
• HF-PIM achieves state-of-the-art verification/recognition on in-the-wild datasets (LFW and IJB-A) when using frontalized results.
• High-resolution (256×256) frontalization on CelebA-HQ is demonstrated, with perceptual and texture details preserved better than prior GAN-based frontalizers.
• The dense correspondence field and texture warping framework enable faithful texture details and pose-invariant reconstruction.
• ARDL enables texture map recovery supervision without 3D data, reducing dependency on labeled 3D datasets.