[論文レビュー] Training Deep Face Recognition Systems with Synthetic Data

Adam Kortylewski, Schneider, Andreas+4 more|arXiv (Cornell University)|Feb 16, 2018

Face recognition and analysis参考文献 31被引用数 51

ひとこと要約

論文は3D Morphable Face Modelからの合成データが深層顔認識を強化し、実データの必要量を減らし、実データでの微調整を行えば、合成データで訓練したモデルと実データで訓練したモデルとのギャップを縮めることができる。

ABSTRACT

Recent advances in deep learning have significantly increased the performance of face recognition systems. The performance and reliability of these models depend heavily on the amount and quality of the training data. However, the collection of annotated large datasets does not scale well and the control over the quality of the data decreases with the size of the dataset. In this work, we explore how synthetically generated data can be used to decrease the number of real-world images needed for training deep face recognition systems. In particular, we make use of a 3D morphable face model for the generation of images with arbitrary amounts of facial identities and with full control over image variations, such as pose, illumination, and background. In our experiments with an off-the-shelf face recognition software we observe the following phenomena: 1) The amount of real training data needed to train competitive deep face recognition systems can be reduced significantly. 2) Combining large-scale real-world data with synthetic data leads to an increased performance. 3) Models trained only on synthetic data with strong variations in pose, illumination, and background perform very well across different datasets even without dataset adaptation. 4) The real-to-virtual performance gap can be closed when using synthetic data for pre-training, followed by fine-tuning with real-world images. 5) There are no observable negative effects of pre-training with synthetic data. Thus, any face recognition system in our experiments benefits from using synthetic face images. The synthetic data generator, as well as all experiments, are publicly available.

研究の動機と目的

• synthetically generated face images can support and improve deep face recognition systems.
• Characterize the real-to-virtual performance gap between synthetic- and real-trained models.
• Demonstrate how synthetic data can reduce real data requirements or augment real data to improve benchmarks.
• Investigate how varying synthetic data properties (pose distribution, identity count) affect real-world performance.
• Provide a reproducible synthetic data generator and analyze limitations of the synthetic-to-real transfer.

提案手法

• Statistically sample a 3D Morphable Model (Basel Face Model) for shape, color, pose, illumination, and expression to generate large-scale synthetic face images.
• Render images with an illumination prior and randomized backgrounds to create realistic variation.
• Train FaceNet-NN4 in the OpenFace framework on synthetic data (SYN-1M) without real-data augmentation or adaptation.
• Measure recognition performance on CMU-Multipie, LFW, and IJB-A using cosine similarity of 128-d embeddings.
• Fine-tune synthetic-pretrained models on varying amounts of real data (Casia subsets) to assess gap closure and performance gains.

実験結果

リサーチクエスチョン

• RQ1 Can synthetic data alone achieve competitive face recognition performance on standard benchmarks?
• RQ2 What is the real-to-virtual gap between models trained on synthetic data versus real data across benchmarks like CMU-Multipie, LFW, and IJB-A?
• RQ3 Can pre-training on synthetic data plus fine-tuning on real data close or bridge the gap to real-data-only models?
• RQ4 How do synthetic data characteristics (pose distribution, number of identities) influence real-world performance?
• RQ5 What is the optimal balance between synthetic pre-training and real-data fine-tuning for best transfer to real-world datasets?

主な発見

• A significant real-to-virtual performance gap exists when training on synthetic data alone, especially on LFW and IJB-A.
• Pre-training on synthetic data followed by fine-tuning with real data closes the gap and can outperform real-data-only models on multiple benchmarks.
• Using synthetic data allows reducing the amount of real data needed to reach competitive performance; e.g., on LFW the gap is reduced with about 100K real images for fine-tuning.
• Combining synthetic data with real data yields performance gains across Multipie, LFW, and IJB-A, often exceeding real data-only baselines.
• The synthetic data generator, based on 200 real 3D scans, can generate diverse identities and variations; pose and background diversity are crucial for transfer gains.
• Increasing the number of identities and maintaining broad pose variation in synthetic data further improves transfer to real datasets.