[Paper Review] Learning Formation of Physically-Based Face Attributes

Ruilong Li, Karl Bladin+9 more|arXiv (Cornell University)|Apr 2, 2020

Face recognition and analysis61 references4 citations

TL;DR

This paper introduces a non-linear, deep learning-based 3D morphable face model trained on 4,000 high-resolution facial scans, enabling pore-level geometric detail and physically-based material attributes (albedo, specular, displacement) at 4K resolution. The model uses a cascading GAN with joint and separate discriminators to generate realistic, anatomically consistent faces, achieving superior expression fitting (1.2mm MSE vs. 2.4mm for linear models) and enabling novel identity generation, mesh fitting, and low-to-high resolution data transfer.

ABSTRACT

Based on a combined data set of 4000 high resolution facial scans, we introduce a non-linear morphable face model, capable of producing multifarious face geometry of pore-level resolution, coupled with material attributes for use in physically-based rendering. We aim to maximize the variety of face identities, while increasing the robustness of correspondence between unique components, including middle-frequency geometry, albedo maps, specular intensity maps and high-frequency displacement details. Our deep learning based generative model learns to correlate albedo and geometry, which ensures the anatomical correctness of the generated assets. We demonstrate potential use of our generative model for novel identity generation, model fitting, interpolation, animation, high fidelity data visualization, and low-to-high resolution data domain transferring. We hope the release of this generative model will encourage further cooperation between all graphics, vision, and data focused professionals while demonstrating the cumulative value of every individual's complete biometric profile.

Motivation & Objective

• To address the lack of high-fidelity, physically-based facial data in public repositories for use in graphics and vision research.
• To overcome limitations of linear 3DMMs in capturing sub-millimeter geometric detail and material consistency.
• To enable controllable generation of diverse, anatomically correct facial identities and expressions using deep generative modeling.
• To democratize access to high-quality facial assets by creating a compact, generative model from a large-scale, high-resolution scan database.
• To support applications such as mesh fitting, low-resolution data enhancement, and high-fidelity visualization in VFX and biometrics.

Proposed method

• A hybrid modeling approach combines non-linear 3DMMs for head geometry with linear PCA for eyes, teeth, and head.
• A cascading generative adversarial network (GAN) is trained to jointly generate geometry and albedo, with separate and joint discriminators ensuring anatomical correctness and texture quality.
• High-resolution 4K texture maps (albedo, specular, displacement) are recovered via inference and upscaling from low-dimensional latent codes.
• An automated pipeline performs dense correspondence and UV parameterization across 26 expressions per subject, enabling consistent alignment and training.
• The model disentangles identity and expression using separate low-dimensional latent vectors, enabling independent control.
• A differentiable rendering and fitting pipeline enables 3D scan registration by inverting the GAN to infer latent codes from target meshes.

Experimental results

Research questions

• RQ1 Can a deep generative model learn to produce high-resolution, physically-based facial attributes (geometry, albedo, specular) from a large-scale, high-fidelity scan database?
• RQ2 How does a non-linear 3DMM compare to linear models in reconstructing facial expressions and fitting to scanned data?
• RQ3 To what extent can the model generalize to novel identities and expressions not seen during training?
• RQ4 Can the model enhance low-resolution facial scans by transferring them into the high-resolution, physically-based domain?
• RQ5 How well can the model support downstream tasks such as 3D mesh fitting and animation retargeting?

Key findings

• The proposed model achieves a mean-square error (MSE) of 1.2mm in expression fitting, outperforming the linear FaceWarehouse model (2.4mm MSE), demonstrating superior reconstruction accuracy.
• The model successfully generates 5,000 novel face identities with gender and age distributions closely aligned to the original training data, indicating strong coverage of the latent space.
• Low-resolution facial scans were effectively enhanced to 4K resolution with realistic material and geometric detail, as shown in qualitative and quantitative comparisons.
• The model enables accurate 3D scan fitting using a non-linear inversion process, outperforming linear models like Basel and FLAME in Hausdorff distance error metrics.
• The joint GAN training with separate and joint discriminators ensures high-quality, anatomically consistent generation of coupled geometry and albedo.
• The model supports novel applications such as identity interpolation, animation, and data domain transfer, demonstrating broad utility in graphics and vision pipelines.