[Paper Review] A high fidelity synthetic face framework for computer vision

Tadas Baltrušaitis, Erroll Wood+7 more|arXiv (Cornell University)|Jul 16, 2020

Face recognition and analysis11 references4 citations

TL;DR

This paper presents a high-fidelity synthetic face framework for computer vision that generates diverse, photorealistic face images with precise ground-truth annotations using a parametric 3D face model, learned identity and expression blendshapes, hand-crafted hair assets, and image-based lighting. The framework enables scalable, consistent, and accurate data synthesis for training vision models with unprecedented control over shape, texture, expression, pose, lighting, and hair.

ABSTRACT

Analysis of faces is one of the core applications of computer vision, with tasks ranging from landmark alignment, head pose estimation, expression recognition, and face recognition among others. However, building reliable methods requires time-consuming data collection and often even more time-consuming manual annotation, which can be unreliable. In our work we propose synthesizing such facial data, including ground truth annotations that would be almost impossible to acquire through manual annotation at the consistency and scale possible through use of synthetic data. We use a parametric face model together with hand crafted assets which enable us to generate training data with unprecedented quality and diversity (varying shape, texture, expression, pose, lighting, and hair).

Motivation & Objective

• To address the challenge of collecting large-scale, accurately annotated real face data for computer vision tasks.
• To generate synthetic face data with ground-truth annotations that are difficult or impossible to obtain manually at scale.
• To enable high-fidelity, diverse, and controllable data synthesis covering identity, expression, pose, lighting, texture, and hair.
• To support machine learning training with realistic, consistent, and fully supervised synthetic face data.
• To provide a scalable pipeline for generating diverse face images with precise control over geometric and appearance factors.

Proposed method

• The framework uses a parametric 3D face model with 7127 vertices and 6908 quad faces, parameterized by identity, expression, and pose using blendshapes and linear-blend skinning.
• Identity and expression blendshapes are learned from 101 high-quality 3D scans using a joint optimization over model parameters and blendshapes to minimize geometric and regularization losses.
• Textures are represented via a VAE with 50-dimensional latent space, trained on cropped face regions using perceptual and MSE losses, enabling disentangled and compact representation.
• Hair is modeled as 3D strands using a physically based shader, with 136 scalp, 50 eyebrows, 71 beards, and 3 eyelash styles, each assigned color via melanin and gray proportion parameters.
• Hair is encoded using UV maps for length, density, and a volume map for flow direction, enabling PCA-based dimensionality reduction for ML compatibility.
• Illumination is simulated using 324 HDR environment maps, preprocessed with log(1+I) and reduced to 50 dimensions via PCA to capture 80% of variance, with 5 random rotations per image for data augmentation.

Experimental results

Research questions

• RQ1 Can a synthetic face generation pipeline produce photorealistic images with accurate, consistent, and fully supervised annotations for computer vision tasks?
• RQ2 How can a 3D face model be optimized to jointly learn identity and expression blendshapes while maintaining geometric fidelity and avoiding artifacts?
• RQ3 To what extent can a VAE-based texture representation preserve facial details while enabling compact, disentangled latent encoding for machine learning?
• RQ4 How can 3D hair strands be modeled and parameterized to support diverse, realistic, and trainable hair styles under varying lighting and face geometry?
• RQ5 Can image-based lighting with PCA-reduced HDR maps effectively simulate diverse, realistic lighting conditions while enabling efficient data augmentation?

Key findings

• The framework generates 1024×1024 photorealistic face images with high diversity in identity, expression, pose, lighting, texture, and hair using a single rendering pipeline.
• The joint optimization of model parameters and identity blendshapes achieves high geometric accuracy, minimizing errors in vertex and normal positions while enforcing anatomical plausibility and mesh quality.
• The VAE-based texture representation achieves a compact 50-dimensional latent space that preserves perceptual quality and enables effective generation of diverse facial textures.
• The hair representation using UV and volume maps enables accurate reconstruction of complex hairstyles with PCA-based dimensionality reduction, supporting efficient ML use.
• The PCA-reduced HDR illumination model captures 80% of variance with only 50 dimensions, enabling efficient and realistic lighting variation across 1620 augmented lighting conditions.
• The entire pipeline produces synthetic data with ground-truth annotations for shape, pose, expression, and appearance, enabling reliable training of computer vision models without manual labeling.