[Paper Review] Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion

David Geissbühler, Hatef Otroshi Shahreza+1 more|arXiv (Cornell University)|Apr 30, 2024

Face recognition and analysis4 citations

TL;DR

This paper proposes a physics-inspired method for generating diverse, high-quality synthetic face datasets by modeling latent space sampling as Brownian motion of soft particles under stochastic forces. Using three algorithms—Langevin, Dispersion, and DisCo—it achieves state-of-the-art face recognition performance (88.40% average accuracy on AgeDB with 30k identities) while minimizing identity leakage, outperforming GAN-based datasets and rivaling diffusion-based methods.

ABSTRACT

Face recognition models are trained on large-scale datasets, which have privacy and ethical concerns. Lately, the use of synthetic data to complement or replace genuine data for the training of face recognition models has been proposed. While promising results have been obtained, it still remains unclear if generative models can yield diverse enough data for such tasks. In this work, we introduce a new method, inspired by the physical motion of soft particles subjected to stochastic Brownian forces, allowing us to sample identities distributions in a latent space under various constraints. We introduce three complementary algorithms, called Langevin, Dispersion, and DisCo, aimed at generating large synthetic face datasets. With this in hands, we generate several face datasets and benchmark them by training face recognition models, showing that data generated with our method exceeds the performance of previously GAN-based datasets and achieves competitive performance with state-of-the-art diffusion-based synthetic datasets. While diffusion models are shown to memorize training data, we prevent leakage in our new synthetic datasets, paving the way for more responsible synthetic datasets.

Motivation & Objective

• To address privacy and ethical concerns in real-world face recognition datasets by generating synthetic alternatives.
• To develop a method that ensures high diversity and realism in synthetic face data while minimizing memorization of training data.
• To enable precise control over inter-class and intra-class variations in synthetic identities through physics-inspired latent space dynamics.
• To benchmark synthetic datasets against state-of-the-art methods in face recognition performance.
• To reduce identity leakage in synthetic data, ensuring responsible and ethical data generation.

Proposed method

• The method models synthetic identities as soft spherical particles in a latent space subject to repulsive forces, Brownian motion, and an attractive potential, inspired by the Langevin equation.
• The Langevin algorithm optimizes inter-class distribution by minimizing embedding distances between synthetic identities while maintaining spread in the latent space.
• The Dispersion algorithm generates intra-class variations by perturbing latent codes with stochastic forces, preserving identity while introducing diversity.
• The DisCo algorithm enhances intra-class variation using pre-computed latent directions, enabling controlled, high-fidelity image generation.
• A hybrid loss function combines identity-level repulsion (L_E), weight regularization (L_W), and genuine-synthetic repulsion (L_E_tr) to prevent data leakage.
• The approach is validated by training face recognition models on generated datasets and evaluating performance on standard benchmarks (LFW, CPLFW, AgeDB, etc.).

Experimental results

Research questions

• RQ1 Can physics-inspired latent space dynamics generate synthetic face datasets with sufficient diversity and realism for training face recognition models?
• RQ2 How does the proposed method compare to GAN-based and diffusion-based synthetic datasets in terms of face recognition accuracy and identity leakage?
• RQ3 To what extent can the Langevin, Dispersion, and DisCo algorithms control inter-class and intra-class variation in synthetic face data?
• RQ4 Does introducing a genuine-synthetic embedding repulsion term (dE_tr) effectively reduce identity leakage in generated datasets?
• RQ5 What is the optimal scale (number of identities and variations) for achieving peak performance in face recognition using synthetic data?

Key findings

• With 30,000 identities and 64 variations per identity using the DisCo algorithm, the method achieves an average face recognition accuracy of 88.40% on the AgeDB benchmark, approaching performance of state-of-the-art diffusion-based methods.
• The method outperforms previously proposed GAN-based synthetic datasets, demonstrating superior generalization and diversity.
• Increasing the genuine-synthetic embedding repulsion distance (dE_tr) from 0 to 1.3 slightly improved face recognition accuracy, suggesting improved generalization.
• Surprisingly, increasing the number of variations to 128 (from 64) caused a significant drop in performance (e.g., average accuracy dropped to 69.32%), indicating overfitting or degradation in quality.
• Visual inspection of the most similar genuine and synthetic image pairs showed minimal identity leakage in the proposed method, unlike DCFace, which exhibited strong memorization of training samples.
• Compounding datasets with reference ensembles provided only marginal performance gains, suggesting that the core algorithms are already effective at scale.