[Paper Review] When Handcrafted Features and Deep Features Meet Mismatched Training and Test Sets for Deepfake Detection

Ying Xu, Sule Yildirim Yayilgan|arXiv (Cornell University)|Sep 27, 2022

Face recognition and analysis4 citations

TL;DR

This paper evaluates handcrafted features (SIFT, HoG) and deep features (Xception, CNN+RNN) for deepfake detection on the FaceForensics++ dataset, revealing that while deep models achieve over 99% accuracy on matched training-test sets, performance plummets when datasets mismatch—highlighting the critical challenge of building a universal deepfake detector.

ABSTRACT

The accelerated growth in synthetic visual media generation and manipulation has now reached the point of raising significant concerns and posing enormous intimidations towards society. There is an imperative need for automatic detection networks towards false digital content and avoid the spread of dangerous artificial information to contend with this threat. In this paper, we utilize and compare two kinds of handcrafted features(SIFT and HoG) and two kinds of deep features(Xception and CNN+RNN) for the deepfake detection task. We also check the performance of these features when there are mismatches between training sets and test sets. Evaluation is performed on the famous FaceForensics++ dataset, which contains four sub-datasets, Deepfakes, Face2Face, FaceSwap and NeuralTextures. The best results are from Xception, where the accuracy could surpass over 99\% when the training and test set are both from the same sub-dataset. In comparison, the results drop dramatically when the training set mismatches the test set. This phenomenon reveals the challenge of creating a universal deepfake detection system.

Motivation & Objective

• To investigate the robustness of handcrafted and deep features in deepfake detection under distribution shifts between training and test sets.
• To compare the performance of SIFT, HoG, Xception, and CNN+RNN on the FaceForensics++ benchmark with varying data distributions.
• To identify the limitations of current deepfake detection models when generalizing across different manipulation methods (e.g., Deepfakes, Face2Face, FaceSwap, NeuralTextures).
• To assess the feasibility of creating a universal deepfake detection system capable of handling diverse synthetic media generation techniques.
• To explore the trade-offs between feature specificity and generalization in deepfake detection under domain shift.

Proposed method

• Extracted handcrafted features using SIFT and Histogram of Oriented Gradients (HoG) from cropped facial regions in video frames.
• Trained a Support Vector Machine (SVM) classifier on handcrafted features for binary deepfake detection.
• Utilized pre-trained Xception and a CNN+RNN architecture to extract deep features from the same facial frames.
• Evaluated models under two settings: matched (same sub-dataset for train and test) and mismatched (different sub-datasets for train and test).
• Used the FaceForensics++ dataset, which includes four sub-datasets: Deepfakes, Face2Face, FaceSwap, and NeuralTextures, each with 1000 original and manipulated videos.
• Performed experiments with a 75% training / 25% test split, with 80% training and 20% validation within the training set.

Experimental results

Research questions

• RQ1 How do handcrafted features (SIFT, HoG) compare to deep features (Xception, CNN+RNN) in deepfake detection accuracy on matched training-test sets?
• RQ2 What is the performance degradation of deepfake detection models when the training and test sets are drawn from different manipulation sub-datasets?
• RQ3 Why do deep models like Xception perform exceptionally well on matched data but fail drastically on mismatched data?
• RQ4 To what extent do handcrafted features generalize across different deepfake generation methods compared to deep features?
• RQ5 Can a universal deepfake detection system be built using current feature extraction and classification approaches under domain shift?

Key findings

• Xception achieved over 99% accuracy when training and test sets were from the same sub-dataset, indicating strong performance on in-domain detection.
• Performance of Xception dropped dramatically on mismatched test sets, with recalls below 5.00% in most cases, showing poor generalization across manipulation types.
• ResNet-152+LSTM achieved 88.53% accuracy on matched data but only around 55% average accuracy on mismatched data, indicating limited robustness.
• Handcrafted features (SIFT, HoG) showed more consistent performance across mismatched sets, with a drop from 74.26% to 56% average accuracy, suggesting better generalization than deep features.
• The recall for Xception on mismatched test sets was less than 5% in most cases, indicating near-total failure to detect deepfakes from unseen manipulation methods.
• The study highlights a fundamental challenge: current deep learning models are highly specialized to specific generation methods and fail to generalize to new ones, undermining the feasibility of universal detection.