[Paper Review] Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection

Jiaming Li, Hongtao Xie+3 more|arXiv (Cornell University)|Mar 16, 2021

Face recognition and analysisComputer Science45 references21 citations

TL;DR

This paper proposes a frequency-aware discriminative feature learning framework (FDFL) for face forgery detection, combining a novel single-center loss (SCL) that enhances intra-class compactness of real faces and inter-class separability, along with an adaptive frequency feature generation module (AFFGM) for data-driven frequency clue mining. The method achieves state-of-the-art performance on all three versions of the FF++ dataset, with a 99.43% accuracy and 0.998 AUC on the most challenging c40 split.

ABSTRACT

Face forgery detection is raising ever-increasing interest in computer vision since facial manipulation technologies cause serious worries. Though recent works have reached sound achievements, there are still unignorable problems: a) learned features supervised by softmax loss are separable but not discriminative enough, since softmax loss does not explicitly encourage intra-class compactness and interclass separability; and b) fixed filter banks and hand-crafted features are insufficient to capture forgery patterns of frequency from diverse inputs. To compensate for such limitations, a novel frequency-aware discriminative feature learning framework is proposed in this paper. Specifically, we design a novel single-center loss (SCL) that only compresses intra-class variations of natural faces while boosting inter-class differences in the embedding space. In such a case, the network can learn more discriminative features with less optimization difficulty. Besides, an adaptive frequency feature generation module is developed to mine frequency clues in a completely data-driven fashion. With the above two modules, the whole framework can learn more discriminative features in an end-to-end manner. Extensive experiments demonstrate the effectiveness and superiority of our framework on three versions of the FF++ dataset.

Motivation & Objective

• To address the limitation of softmax loss in learning discriminative features due to insufficient explicit constraints on intra-class compactness and inter-class separability.
• To overcome the reliance on fixed filter banks and hand-crafted features in frequency-based forgery detection, which limits adaptability to diverse forgery patterns.
• To develop a data-driven frequency feature extraction mechanism that captures subtle, forgery-specific artifacts across varying manipulation methods.
• To improve generalization and robustness in face forgery detection under different compression levels and unseen manipulation techniques.

Proposed method

• Introduces a single-center loss (SCL) that minimizes the distance between real face embeddings and a single center point while enforcing a margin between real and fake face distances to the center.
• SCL explicitly promotes intra-class compactness only for real faces, avoiding over-constraint on diverse fake face distributions, thus reducing optimization difficulty.
• Designs an adaptive frequency feature generation module (AFFGM) that learns frequency-specific representations through a learnable frequency preprocessing stage and an adaptive frequency information mining block.
• The AFFGM enables end-to-end learning of frequency-domain artifacts without relying on fixed filter banks, allowing the model to discover relevant frequency patterns from data.
• Combines SCL and AFFGM within a unified end-to-end training framework to jointly optimize discriminative feature learning and frequency-aware representation.
• Employs a Xception backbone for feature extraction, with SCL and AFFGM integrated as loss and feature engineering components respectively.

Figure 1: The feature distribution of samples in the embedding space. Left: learned features supervised by softmax loss are broadly separable but not discriminative enough, since the intra-class compactness and inter-class separability are not explicitly constrained. Right: our SCL only encourages t

Experimental results

Research questions

• RQ1 Can a metric learning loss that only constrains intra-class compactness for real faces improve feature discriminability and reduce optimization difficulty in face forgery detection?
• RQ2 Can a data-driven, adaptive frequency feature generation module outperform hand-crafted or fixed filter-based frequency analysis in capturing subtle forgery artifacts?
• RQ3 How does the proposed FDFL framework perform across different levels of video compression (c0, c23, c40) on the FF++ dataset?
• RQ4 To what extent does the integration of SCL and AFFGM improve detection performance compared to baseline models and prior state-of-the-art methods?
• RQ5 Does the framework generalize to unseen manipulation methods, and what are its limitations in such scenarios?

Key findings

• The proposed FDFL framework achieves a 99.43% accuracy, 0.998 AUC, and 0.924 pAUC 0.1 on the most challenging c40 version of the FF++ dataset, outperforming prior state-of-the-art methods.
• The single-center loss (SCL) alone improves AUC by 5.5% and pAUC 0.1 by 13.8% over the baseline, demonstrating its effectiveness in enhancing feature discriminability.
• The adaptive frequency feature generation module (AFFGM) contributes a 4.5% improvement in AUC and 11.7% in pAUC 0.1, showing its value in capturing frequency-domain artifacts.
• When combined, SCL and AFFGM yield the best performance (AUC: 0.924, pAUC 0.1: 0.810), confirming their complementary nature and synergy in the framework.
• On the c40 split, FDFL achieves a 6.4% AUC gain, 15.8% pAUC 0.1 gain, and 2.86% accuracy gain over the previous SOTA method, indicating strong robustness to heavy compression.
• The ablation study confirms that both SCL and AFFGM independently contribute significantly to performance, with the full model achieving the highest scores across all metrics.

Figure 2: The Frequency-aware Discriminative Features Learning framework. AFFGM stands for the adaptive frequency feature generation module. AFIMB represents the adaptive frequency information mining block. FC represents the fully connected layer and $L_{ce}$ represents the cross-entropy loss. The w