[Paper Review] Training Deep Face Recognition Systems with Synthetic Data

Adam Kortylewski, Schneider, Andreas+4 more|arXiv (Cornell University)|Feb 16, 2018

Face recognition and analysis31 references51 citations

TL;DR

The paper shows synthetic data from a 3D Morphable Face Model can bolster deep face recognition, reduce real data needs, and, with fine-tuning on real data, close the gap between synthetic-trained and real-trained models.

ABSTRACT

Recent advances in deep learning have significantly increased the performance of face recognition systems. The performance and reliability of these models depend heavily on the amount and quality of the training data. However, the collection of annotated large datasets does not scale well and the control over the quality of the data decreases with the size of the dataset. In this work, we explore how synthetically generated data can be used to decrease the number of real-world images needed for training deep face recognition systems. In particular, we make use of a 3D morphable face model for the generation of images with arbitrary amounts of facial identities and with full control over image variations, such as pose, illumination, and background. In our experiments with an off-the-shelf face recognition software we observe the following phenomena: 1) The amount of real training data needed to train competitive deep face recognition systems can be reduced significantly. 2) Combining large-scale real-world data with synthetic data leads to an increased performance. 3) Models trained only on synthetic data with strong variations in pose, illumination, and background perform very well across different datasets even without dataset adaptation. 4) The real-to-virtual performance gap can be closed when using synthetic data for pre-training, followed by fine-tuning with real-world images. 5) There are no observable negative effects of pre-training with synthetic data. Thus, any face recognition system in our experiments benefits from using synthetic face images. The synthetic data generator, as well as all experiments, are publicly available.

Motivation & Objective

• Motivate and quantify whether synthetically generated face images can support and improve deep face recognition systems.
• Characterize the real-to-virtual performance gap between synthetic- and real-trained models.
• Demonstrate how synthetic data can reduce real data requirements or augment real data to improve benchmarks.
• Investigate how varying synthetic data properties (pose distribution, identity count) affect real-world performance.
• Provide a reproducible synthetic data generator and analyze limitations of the synthetic-to-real transfer.

Proposed method

• Generate large-scale synthetic face images by sampling a statistical 3D Morphable Model (Basel Face Model) for shape, color, pose, illumination, and expression.
• Render images with an illumination prior and randomized backgrounds to create realistic variation.
• Train FaceNet-NN4 in the OpenFace framework on synthetic data (SYN-1M) without real-data augmentation or adaptation.
• Measure recognition performance on CMU-Multipie, LFW, and IJB-A using cosine similarity of 128-d embeddings.
• Fine-tune synthetic-pretrained models on varying amounts of real data (Casia subsets) to assess gap closure and performance gains.

Experimental results

Research questions

• RQ1 Can synthetic data alone achieve competitive face recognition performance on standard benchmarks?
• RQ2 What is the real-to-virtual gap between models trained on synthetic data versus real data across benchmarks like CMU-Multipie, LFW, and IJB-A?
• RQ3 Can pre-training on synthetic data plus fine-tuning on real data close or bridge the gap to real-data-only models?
• RQ4 How do synthetic data characteristics (pose distribution, number of identities) influence real-world performance?
• RQ5 What is the optimal balance between synthetic pre-training and real-data fine-tuning for best transfer to real-world datasets?

Key findings

• A significant real-to-virtual performance gap exists when training on synthetic data alone, especially on LFW and IJB-A.
• Pre-training on synthetic data followed by fine-tuning with real data closes the gap and can outperform real-data-only models on multiple benchmarks.
• Using synthetic data allows reducing the amount of real data needed to reach competitive performance; e.g., on LFW the gap is reduced with about 100K real images for fine-tuning.
• Combining synthetic data with real data yields performance gains across Multipie, LFW, and IJB-A, often exceeding real data-only baselines.
• The synthetic data generator, based on 200 real 3D scans, can generate diverse identities and variations; pose and background diversity are crucial for transfer gains.
• Increasing the number of identities and maintaining broad pose variation in synthetic data further improves transfer to real datasets.