[Paper Review] Generated Faces in the Wild: Quantitative Comparison of Stable Diffusion, Midjourney and DALL-E 2

Ali Borji|arXiv (Cornell University)|Oct 2, 2022

Generative Adversarial Networks and Image SynthesisComputer Science96 citations

TL;DR

The paper quantitatively compares photorealistic face generation in the wild across Stable Diffusion, Midjourney, and DALL-E 2 using FID; Stable Diffusion yields the best face quality among the three. A new GFW dataset of generated faces is introduced for evaluation.

ABSTRACT

The field of image synthesis has made great strides in the last couple of years. Recent models are capable of generating images with astonishing quality. Fine-grained evaluation of these models on some interesting categories such as faces is still missing. Here, we conduct a quantitative comparison of three popular systems including Stable Diffusion, Midjourney, and DALL-E 2 in their ability to generate photorealistic faces in the wild. We find that Stable Diffusion generates better faces than the other systems, according to the FID score. We also introduce a dataset of generated faces in the wild dubbed GFW, including a total of 15,076 faces. Furthermore, we hope that our study spurs follow-up research in assessing the generative models and improving them. Data and code are available at data and code, respectively.

Motivation & Objective

• Evaluate the ability of three text-to-image models to generate photorealistic faces in cluttered scenes.
• Provide a fair quantitative comparison using a controlled face dataset and FID as the evaluation metric.
• Create a dataset of generated faces to enable future research on generative model evaluation and biases.

Proposed method

• Use COCO captions to prompt three models to generate images containing faces.
• Detect faces in generated and real images using MediaPipe face detector and prune false positives.
• Resize generated faces to 100x100 and compute Fréchet Inception Distance (FID) against real faces; repeat across multiple random samples to estimate mean and std.

Experimental results

Research questions

• RQ1 Can Stable Diffusion, Midjourney, and DALL-E 2 produce photorealistic faces in cluttered scenes, and how do they compare quantitatively?
• RQ2 Does FID reliably distinguish face quality across models given differing sample sizes and potential memorization or safeguards?
• RQ3 What are the practical limitations and failure modes for generated faces in the wild across these systems?

Key findings

• Stable Diffusion yields the best FID among the three models for generated faces.
• All models struggle with eyeglasses, eyes, occlusions, profile views, and facial symmetry, with real faces remaining markedly higher quality.
• DALL-E 2 performs worse than Stable Diffusion, potentially due to safeguards, focus on portraits, and smaller image sets; Midjourney often produces surreal or anime-like faces.
• A larger sample size improves stability of FID comparisons, and the authors created a 15,076-face generated set (8,050 Stable Diffusion, 6,350 Midjourney, 676 DALL-E 2).
• The study highlights potential memorization and watermark issues, and suggests broader evaluation metrics beyond FID (e.g., SSIM, LPIPS, human judgments).
• The paper provides data and code access to facilitate future benchmarking and deeper analysis of face generation in the wild.