[Paper Review] Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues

Yu-Yang Qian, Guojun Yin+3 more|arXiv (Cornell University)|Jul 18, 2020

Digital Media Forensic DetectionComputer Science56 references42 citations

TL;DR

The paper introduces F3-Net, a frequency-aware two-stream framework that uses Frequency-aware Decomposition (FAD) and Local Frequency Statistics (LFS) with a MixBlock cross-attention module to detect face forgeries, achieving state-of-the-art results on FaceForensics++ especially under low-quality compression.

ABSTRACT

As realistic facial manipulation technologies have achieved remarkable progress, social concerns about potential malicious abuse of these technologies bring out an emerging research topic of face forgery detection. However, it is extremely challenging since recent advances are able to forge faces beyond the perception ability of human eyes, especially in compressed images and videos. We find that mining forgery patterns with the awareness of frequency could be a cure, as frequency provides a complementary viewpoint where either subtle forgery artifacts or compression errors could be well described. To introduce frequency into the face forgery detection, we propose a novel Frequency in Face Forgery Network (F3-Net), taking advantages of two different but complementary frequency-aware clues, 1) frequency-aware decomposed image components, and 2) local frequency statistics, to deeply mine the forgery patterns via our two-stream collaborative learning framework. We apply DCT as the applied frequency-domain transformation. Through comprehensive studies, we show that the proposed F3-Net significantly outperforms competing state-of-the-art methods on all compression qualities in the challenging FaceForensics++ dataset, especially wins a big lead upon low-quality media.

Motivation & Objective

• Motivate robust face forgery detection under compression and subtle artifacts that are hard to see in RGB space.
• Introduce frequency-aware clues that capture high-frequency artifacts and compression errors.
• Propose a two-stream architecture that collaboratively learns from frequency-decomposed components and local frequency statistics.

Proposed method

• Propose Frequency-aware Decomposition (FAD) that partitions the frequency domain into learnable bands and reconstructs multiple frequency-aware image components.
• Propose Local Frequency Statistics (LFS) by applying Sliding Window DCT (SWDCT) to extract localized frequency responses and aggregating mean responses per band.
• Use a two-stream backbone (Xception) for FAD and LFS, connected via a cross-attention MixBlock for progressive fusion.
• Train end-to-end with cross-entropy loss, evaluating on FaceForensics++ across LQ/HQ/RAW settings.

Experimental results

Research questions

• RQ1 Can frequency-aware cues improve forgery detection performance under varying compression qualities?
• RQ2 Do frequency-aware decomposed components and local frequency statistics provide complementary information that a fusion module can leverage for better detection?
• RQ3 Does cross-attention-based fusion (MixBlock) enhance two-stream interaction to outperform RGB-based baselines?
• RQ4 How does the proposed method perform across different manipulation types in FaceForensics++?
• RQ5 Is the approach transferable to video-level frameworks beyond single-frame analysis?

Key findings

• F3-Net outperforms state-of-the-art methods on FaceForensics++ across LQ, HQ, and RAW settings, with notable gains in low-quality media.
• Ablation studies show FAD (frequency-aware decomposition) and LFS (local frequency statistics) each contribute to performance, with MixBlock providing further gains.
• High-frequency components are particularly informative for forgery detection; using all frequency bands yields the best results.
• The two-stream framework with cross-attention fusion yields better ROC characteristics, especially at low false-positive rates.
• Video extension with SlowFast backbone further improves results, achieving top performance on several metrics.