[Paper Review] DeepFaceLab: Integrated, flexible and extensible face-swapping framework

Ivan Perov, Daiheng Gao+12 more|arXiv (Cornell University)|May 12, 2020

Generative Adversarial Networks and Image Synthesis25 references87 citations

TL;DR

DeepFaceLab (DFL) is an open-source, integrated framework for high-quality, cinema-grade face-swapping with a flexible, modular pipeline and extensive extraction, training, and conversion capabilities.

ABSTRACT

Deepfake defense not only requires the research of detection but also requires the efforts of generation methods. However, current deepfake methods suffer the effects of obscure workflow and poor performance. To solve this problem, we present DeepFaceLab, the current dominant deepfake framework for face-swapping. It provides the necessary tools as well as an easy-to-use way to conduct high-quality face-swapping. It also offers a flexible and loose coupling structure for people who need to strengthen their pipeline with other features without writing complicated boilerplate code. We detail the principles that drive the implementation of DeepFaceLab and introduce its pipeline, through which every aspect of the pipeline can be modified painlessly by users to achieve their customization purpose. It is noteworthy that DeepFaceLab could achieve cinema-quality results with high fidelity. We demonstrate the advantage of our system by comparing our approach with other face-swapping methods.For more information, please visit:https://github.com/iperov/DeepFaceLab/.

Motivation & Objective

• Provide an integrated, user-friendly pipeline for high-fidelity face swapping.
• Enable flexible customization by modular, interchangeable components.
• Support large-scale datasets and scalable training for cinema-quality results.
• Offer tools to aid forgery detection by producing high-quality synthetic data.

Proposed method

• Three-phase pipeline: extraction, training, and conversion with a loose-coupling design.
• Extraction: face detection, alignment, and segmentation with multiple modes and canonical landmark templates; optional XSeg few-shot segmentation customization.
• Training: two structures (DF and LIAE) with shared encoders and either shared or separate interaction modules; loss functions combining DSSIM and MSE with weighted masks; TrueFace and GAN variants to improve realism.
• Conversion: repositioning via Umeyama-based alignment, color transfer options, Poisson blending, and super-resolution sharpening to enhance realism.
• Extensibility: interchangeable components (detectors, segmenters) and tools (XSeg) for customization and improved robustness.

Experimental results

Research questions

• RQ1 How can a unified, configurable pipeline improve the quality and ease of performing face-swapping tasks?
• RQ2 To what extent do different model structures (DF vs. LIAE) and training paradigms affect fidelity, pose preservation, and identity retention?
• RQ3 Can large-scale datasets and advanced post-processing yield cinema-quality results in face-swapping?
• RQ4 What is the impact of modular components (XSeg, different detectors/segmenters) on robustness to occlusions and varied poses?

Key findings

• DFL achieves competitive qualitative and quantitative results against state-of-the-art face-swapping methods.
• In Table 1, DFL shows higher SSIM and lower perceptual loss and pose/landmark errors compared with baselines, indicating better structure similarity and stability.
• Ablation studies show LIAE structures better inherit target face shape and, with GAN and TrueFace, yield more photorealistic details and stable results.
• GAN-based training improves detail realism such as wrinkles and moles, while TrueFace stabilizes training to preserve identity and quality.
• XSeg and TernausNet-based segmentation enhance handling of occlusions and allow customization for specific faces.