[Paper Review] Protecting President Zelenskyy against Deep Fakes

Matyáš Boháček, Hany Farid|arXiv (Cornell University)|Jun 24, 2022

Face Recognition and Perception5 citations

TL;DR

This paper proposes an identity-based deepfake detection system for Ukrainian President Zelenskyy using facial and gestural behavioral features extracted from over eight hours of authentic video. By modeling distinctive head and hand movements, the method achieves 99.88% accuracy in distinguishing real Zelenskyy from deepfake imitators, offering a robust defense against disinformation in conflict zones.

ABSTRACT

The 2022 Russian invasion of Ukraine is being fought on two fronts: a brutal ground war and a duplicitous disinformation campaign designed to conceal and justify Russia's actions. This campaign includes at least one example of a deep-fake video purportedly showing Ukrainian President Zelenskyy admitting defeat and surrendering. In anticipation of future attacks of this form, we describe a facial and gestural behavioral model that captures distinctive characteristics of Zelenskyy's speaking style. Trained on over eight hours of authentic video from four different settings, we show that this behavioral model can distinguish Zelenskyy from deep-fake imposters.This model can play an important role -- particularly during the fog of war -- in distinguishing the real from the fake.

Motivation & Objective

• To address the threat of deepfake videos targeting world leaders during wartime, particularly in the context of Russia’s disinformation campaign during the invasion of Ukraine.
• To develop a robust, identity-specific detection system that leverages behavioral biometrics to distinguish real from synthetic media of a single high-value individual.
• To enhance traditional facial-based deepfake detection by incorporating dynamic gestural features, improving discriminative power and resilience to spoofing.
• To create a practical, deployable defense mechanism for protecting key political figures from high-impact disinformation during crises.

Proposed method

• Collected and analyzed 506 minutes of authentic video footage of Zelenskyy across four distinct contexts: public addresses, press briefings, bunker appearances, and informal interviews.
• Extracted 496 facial and gestural features, including 3D head pose (Rx, Ry, Rz), hand joint positions (elbow, wrist, shoulder), and facial action units (AUs) from 2D and 3D facial landmarks.
• Trained a binary classifier using a combination of facial and gestural features to distinguish real Zelenskyy from deepfake imitators and other world leaders.
• Evaluated model performance using a holdout test set including real Zelenskyy, deepfake Zelenskyy, and videos of other world leaders to assess generalization and specificity.
• Identified the most discriminative feature pairs through 500 random feature subset trials, focusing on correlations between head motion and hand gestures.
• Used median accuracy across 500 classifiers to rank feature importance, revealing that head-pose and hand movement correlations are the most predictive.

Experimental results

Research questions

• RQ1 Can a behaviorally grounded, identity-specific model effectively detect deepfake videos of President Zelenskyy using only his authentic video footage?
• RQ2 How do facial mannerisms and gestural features collectively improve deepfake detection accuracy compared to facial features alone?
• RQ3 To what extent do the most discriminative features—particularly correlations between head and hand movements—contribute to detection performance?
• RQ4 How does the model generalize when tested on videos of other world leaders and deepfake Zelenskyy?
• RQ5 What is the minimum number of features required to achieve high detection accuracy, and how does performance scale with feature set size?

Key findings

• The model achieves a detection accuracy of 99.88% when using all 496 facial and gestural features, demonstrating near-perfect discrimination between real and fake Zelenskyy.
• With 400 features, the model reaches 99.52% accuracy, indicating that performance plateaus after a moderate feature set size, suggesting high efficiency in feature utilization.
• The top 10 most discriminative feature pairs—such as head-pose-Rx with right-elbow-y—achieve a classifier accuracy of 44.4%, significantly outperforming random feature subsets (median 8.4%).
• The correlation between head rotation (e.g., nodding) and hand gestures is the most predictive signal, highlighting the importance of multimodal behavioral modeling.
• A single classifier using only the top 20 features achieves 63.4% accuracy, underscoring the necessity of a comprehensive feature set for high performance.
• The model maintains strong performance across diverse contexts, indicating that the behavioral model generalizes well across different speaking styles and settings.