[Paper Review] FakeLocator: Robust Localization of GAN-Based Face Manipulations

Yihao Huang, Felix Juefei-Xu+3 more|arXiv (Cornell University)|Jan 27, 2020

Digital Media Forensic Detection6 citations

TL;DR

FakeLocator proposes a novel, robust method for high-resolution localization of GAN-generated face manipulations by exploiting upsampling artifacts through a grayscale fakeness map. It achieves state-of-the-art performance across multiple datasets and GAN methods, with strong robustness to real-world degradations like JPEG compression and blur.

ABSTRACT

Full face synthesis and partial face manipulation by virtue of the generative adversarial networks (GANs) and its variants have raised wide public concerns. In the multi-media forensics area, detecting and ultimately locating the image forgery has become an imperative task. In this work, we investigate the architecture of existing GAN-based face manipulation methods and observe that the imperfection of upsampling methods therewithin could be served as an important asset for GAN-synthesized fake image detection and forgery localization. Based on this basic observation, we have proposed a novel approach, termed FakeLocator, to obtain high localization accuracy, at full resolution, on manipulated facial images. To the best of our knowledge, this is the very first attempt to solve the GAN-based fake localization problem with a gray-scale fakeness map that preserves more information of fake regions. To improve the universality of FakeLocator across multifarious facial attributes, we introduce an attention mechanism to guide the training of the model. To improve the universality of FakeLocator across different DeepFake methods, we propose partial data augmentation and single sample clustering on the training images. Experimental results on popular FaceForensics++, DFFD datasets and seven different state-of-the-art GAN-based face generation methods have shown the effectiveness of our method. Compared with the baselines, our method performs better on various metrics. Moreover, the proposed method is robust against various real-world facial image degradations such as JPEG compression, low-resolution, noise, and blur.

Motivation & Objective

• To address the critical need for precise localization of fake facial regions in GAN-generated images, beyond binary detection.
• To overcome limitations in existing methods that fail to preserve fine-grained fake region details due to low-resolution outputs or lossy feature extraction.
• To improve model generalization across diverse facial attributes (e.g., hair color, gender) and multiple GAN-based face manipulation methods.
• To ensure robustness against real-world image degradations such as JPEG compression, noise, blur, and low-resolution scaling.

Proposed method

• Leverages the inherent imperfections in upsampling layers of GAN-based face synthesis models as a detectable fingerprint for forgery localization.
• Introduces a grayscale fakeness map as the output, preserving high-resolution spatial details of fake regions without post-processing.
• Incorporates an attention mechanism guided by face parsing maps to enhance cross-attribute generalization across diverse facial attributes.
• Applies partial data augmentation and single-sample clustering during training to improve cross-method generalization across different GAN architectures.
• Employs an encoder-decoder architecture with skip connections to maintain spatial resolution and reconstruct fine-grained forgery patterns.
• Uses ground truth fakeness maps computed as the absolute difference between real and fake images to supervise training.

Experimental results

Research questions

• RQ1 Can upsampling artifacts in GAN-based face manipulation models be reliably exploited for high-resolution forgery localization?
• RQ2 How can a deep learning model be made robust to variations in facial attributes (e.g., hair color, gender) while maintaining high localization accuracy?
• RQ3 To what extent can a single model generalize across diverse GAN-based face generation methods without retraining?
• RQ4 How does the proposed method perform under real-world image degradations such as JPEG compression, blur, noise, and low-resolution scaling?

Key findings

• FakeLocator achieves state-of-the-art performance on the FaceForensics++ and DFFD datasets, outperforming existing methods across multiple metrics including PSNR, SSIM, and ACC.
• The model maintains high localization accuracy even after severe image degradations: PSNR remains stable under JPEG compression and noise, while SSIM and ACC show only minor degradation under extreme conditions.
• The disorganization test confirms the model’s location-independence, as performance drops only slightly when fake image patches are rearranged, indicating it learns texture-based patterns rather than spatial cues.
• The attention mechanism guided by face parsing significantly improves cross-attribute generalization, especially for rare or complex attributes like 'bald' or 'beard'.
• Partial data augmentation and single-sample clustering enhance cross-method robustness, enabling the model to generalize across seven different SOTA GAN-based face manipulation methods.
• The grayscale fakeness map output preserves fine-grained details, enabling precise localization of subtle manipulations such as hair color or eye shape changes.