[Paper Review] Generation of Non-Deterministic Synthetic Face Datasets Guided by Identity Priors

Marcel Grimmer, Haoyu Zhang+3 more|arXiv (Cornell University)|Dec 7, 2021

Face recognition and analysis4 citations

TL;DR

This paper proposes a non-deterministic method for generating synthetic mated face images using PCA-guided latent space manipulation in StyleGAN, with dynamic identity preservation via a pre-trained face recognition system. The approach produces the SymFace dataset (77,034 images, 25,919 IDs), demonstrating that synthetic mated samples yield well-separated genuine and impostor score distributions comparable to real biometric data, making them viable for biometric performance testing while avoiding privacy and bias issues.

ABSTRACT

Enabling highly secure applications (such as border crossing) with face recognition requires extensive biometric performance tests through large scale data. However, using real face images raises concerns about privacy as the laws do not allow the images to be used for other purposes than originally intended. Using representative and subsets of face data can also lead to unwanted demographic biases and cause an imbalance in datasets. One possible solution to overcome these issues is to replace real face images with synthetically generated samples. While generating synthetic images has benefited from recent advancements in computer vision, generating multiple samples of the same synthetic identity resembling real-world variations is still unaddressed, i.e., mated samples. This work proposes a non-deterministic method for generating mated face images by exploiting the well-structured latent space of StyleGAN. Mated samples are generated by manipulating latent vectors, and more precisely, we exploit Principal Component Analysis (PCA) to define semantically meaningful directions in the latent space and control the similarity between the original and the mated samples using a pre-trained face recognition system. We create a new dataset of synthetic face images (SymFace) consisting of 77,034 samples including 25,919 synthetic IDs. Through our analysis using well-established face image quality metrics, we demonstrate the differences in the biometric quality of synthetic samples mimicking characteristics of real biometric data. The analysis and results thereof indicate the use of synthetic samples created using the proposed approach as a viable alternative to replacing real biometric data.

Motivation & Objective

• To address privacy and demographic bias concerns in real biometric datasets used for face recognition testing.
• To generate synthetic face images that replicate intra-identity variations found in real-world biometric data, such as pose, expression, and illumination changes.
• To ensure identity preservation during synthetic image generation despite non-deterministic sampling in the latent space.
• To create a large-scale, privacy-compliant synthetic dataset suitable for biometric performance evaluation.

Proposed method

• Utilizes a pre-trained StyleGAN generator to produce base synthetic face images represented by 512-dimensional latent vectors.
• Applies Principal Component Analysis (PCA) to the latent space of 50,000 generated images to identify semantically meaningful directions for editing.
• Generates mated samples by perturbing the latent vectors along the top principal components, introducing realistic variations in facial attributes.
• Employs a pre-trained face recognition model (FR) to dynamically monitor and control the similarity between original and edited images, preserving identity.
• Uses a feedback loop where the FR system evaluates the cosine similarity between original and generated images to guide the perturbation magnitude.
• Constructs the SymFace dataset with 77,034 images across 25,919 synthetic identities, including variations in yaw, illumination, and smile.

Experimental results

Research questions

• RQ1 Can non-deterministic generation of mated face images be achieved while preserving identity fidelity using latent space manipulation in StyleGAN?
• RQ2 To what extent do PCA-derived directions in the latent space produce semantically meaningful facial variations resembling real biometric data?
• RQ3 How do the biometric comparison score distributions of synthetic mated samples compare to those of real bona fide datasets like FRGC v2.0?
• RQ4 Can synthetic mated samples generate well-separated genuine and impostor score distributions suitable for performance testing?
• RQ5 How do face quality metrics of synthetic images compare to real-world benchmarks?

Key findings

• The PCA-FR-Guided sampling method produced mated comparison score distributions that are well-separated from non-mated scores, with a clear distinction visible in the kernel density curves.
• The Kullback-Leibler divergence between the mated score distribution of the proposed method and FRGC v2.0 was 0.42, indicating a moderate but acceptable similarity in genuine score patterns.
• The synthetic dataset achieved a KL divergence of 0.17 with FRGC v2.0 on smile-edited samples, suggesting strong alignment in intra-class variation for this attribute.
• Face quality assessment metrics (SER-FIQ and FaceQnet v1) showed that the synthetic images' quality distributions were comparable to real data, with SER-FIQ scores of 1.17 for PCA-FR and 1.19 for illumination editing.
• The non-mated comparison score distributions of all synthetic datasets remained well below the FMR=0.1% threshold, indicating low false match risk and minimal lookalike generation.
• Editing yaw angles resulted in the widest span of mated comparison scores among all attributes, suggesting higher variability in pose-related edits.