[Paper Review] DCFace: Synthetic Face Generation with Dual Condition Diffusion Model

Minchul Kim, Feng Liu+2 more|arXiv (Cornell University)|Apr 14, 2023

Face recognition and analysis4 citations

TL;DR

This paper proposes DCFace, a dual condition diffusion model that generates high-fidelity synthetic face images by separately controlling identity (ID) and style (e.g., pose, lighting, expression), enabling consistent generation of multiple images per subject under diverse styles. Trained on DCFace-generated data, face recognition models achieve 6.11% higher average accuracy than prior methods on four of five benchmark datasets, significantly narrowing the performance gap to real-data training.

ABSTRACT

Generating synthetic datasets for training face recognition models is challenging because dataset generation entails more than creating high fidelity images. It involves generating multiple images of same subjects under different factors ( extit{e.g.}, variations in pose, illumination, expression, aging and occlusion) which follows the real image conditional distribution. Previous works have studied the generation of synthetic datasets using GAN or 3D models. In this work, we approach the problem from the aspect of combining subject appearance (ID) and external factor (style) conditions. These two conditions provide a direct way to control the inter-class and intra-class variations. To this end, we propose a Dual Condition Face Generator (DCFace) based on a diffusion model. Our novel Patch-wise style extractor and Time-step dependent ID loss enables DCFace to consistently produce face images of the same subject under different styles with precise control. Face recognition models trained on synthetic images from the proposed DCFace provide higher verification accuracies compared to previous works by $6.11\%$ on average in $4$ out of $5$ test datasets, LFW, CFP-FP, CPLFW, AgeDB and CALFW. Code is available at https://github.com/mk-minchul/dcface

Motivation & Objective

• To address the challenge of generating large-scale, label-consistent, and diverse synthetic face datasets for training face recognition models.
• To improve upon existing GAN- and 3D model-based methods by jointly controlling identity and style factors in image generation.
• To achieve high face recognition performance using only synthetic data, reducing reliance on privacy-invasive web-crawled datasets.
• To balance inter-class variation (diverse identities), intra-class variation (multiple styles per identity), and label consistency in synthetic data.
• To close the performance gap between synthetic and real training data in face recognition benchmarks.

Proposed method

• A two-stage generation framework: first, generate high-quality ID-specific face images using a diffusion model; second, mix these with style images from a style bank via a dual-condition generator.
• Introduce a patch-wise style extractor that disentangles style information (pose, lighting, expression) from identity, enabling fine-grained style control.
• Propose a time-step dependent ID loss that preserves identity consistency across diffusion sampling steps, ensuring the generated image retains the source subject's identity.
• Train the dual-condition generator using only real (ID, style) pairs from the same subject, avoiding the need for hard-to-achieve cross-condition triplets.
• Leverage a style bank containing diverse style images to enable large-scale style variation while maintaining identity fidelity.
• Apply label oversampling during training to compensate for lower label consistency in synthetic data, improving model generalization.

Experimental results

Research questions

• RQ1 Can a diffusion-based model effectively generate diverse, identity-consistent face images by conditioning on both identity and style?
• RQ2 How does controlling identity and style separately improve the quality and diversity of synthetic face datasets compared to prior GAN- or 3D-based methods?
• RQ3 To what extent can face recognition models trained exclusively on synthetic data match the performance of models trained on real data?
• RQ4 How does the proposed dual-condition generation framework reduce the performance gap between synthetic and real training data?
• RQ5 What trade-offs exist between label consistency, style diversity, and identity fidelity in synthetic face generation?

Key findings

• DCFace-generated synthetic datasets achieve an average verification accuracy improvement of 6.11% over previous state-of-the-art methods on four out of five benchmark datasets (LFW, CFP-FP, CPLFW, AgeDB, CALFW).
• On the 0.5M image setting, DCFace reduces the performance gap to real data by 57% compared to the previous SoTA, with a synthetic-to-real accuracy gap of 4.14%.
• In the 1.2M image setting, the gap to real data is further reduced to 3.74%, a 60.9% improvement over the previous method (9.55% gap).
• The model generates over 1 million images with 20,000 unique identities, each with 50 diverse style variations, demonstrating high scalability and diversity.
• The time-step dependent ID loss and patch-wise style extractor significantly enhance identity consistency and style disentanglement, as validated in ablation studies.
• Label oversampling during training improves performance by 0.52% over random sampling, confirming the benefit of compensating for synthetic data's lower label consistency.