[Paper Review] A Cross-Verification Approach for Protecting World Leaders from Fake and Tampered Audio

Mengyi Shan, T. J. Tsai|arXiv (Cornell University)|Oct 23, 2020

Digital Media Forensic Detection26 references4 citations

TL;DR

This paper proposes a cross-verification framework that authenticates world leaders' speech recordings by aligning suspicious audio against trusted reference recordings using a modified Needleman-Wunsch algorithm and classifying frame-level matches with deep learning models. It achieves 99.7% alignment accuracy at 50 ms tolerance and a 0.43% equal error rate in frame attribution, significantly outperforming dynamic time warping and existing methods in detecting tampered audio.

ABSTRACT

This paper tackles the problem of verifying the authenticity of speech recordings from world leaders. Whereas previous work on detecting deep fake or tampered audio focus on scrutinizing an audio recording in isolation, we instead reframe the problem and focus on cross-verifying a questionable recording against trusted references. We present a method for cross-verifying a speech recording against a reference that consists of two steps: aligning the two recordings and then classifying each query frame as matching or non-matching. We propose a subsequence alignment method based on the Needleman-Wunsch algorithm and show that it significantly outperforms dynamic time warping in handling common tampering operations. We also explore several binary classification models based on LSTM and Transformer architectures to verify content at the frame level. Through extensive experiments on tampered speech recordings of Donald Trump, we show that our system can reliably detect audio tampering operations of different types and durations. Our best model achieves 99.7% accuracy for the alignment task at an error tolerance of 50 ms and a 0.43% equal error rate in classifying audio frames as matching or non-matching.

Motivation & Objective

• To address the growing threat of deepfake and tampered audio targeting world leaders.
• To develop a robust method for verifying speech authenticity using trusted reference recordings instead of analyzing audio in isolation.
• To enable positive verification of genuine recordings, not just detection of fakes.
• To achieve high accuracy in detecting various tampering operations, including insertions, deletions, and replacements.
• To create a system resilient to evolving audio spoofing technologies by relying on cross-verification with multiple sources.

Proposed method

• A two-stage cross-verification system: first aligning the query audio with a trusted reference using a modified Needleman-Wunsch algorithm (SNWTW).
• SNWTW introduces a skip penalty independent of frame mismatch costs, enabling better handling of insertions and deletions compared to dynamic time warping.
• The alignment process uses a scoring matrix with weights for matches, mismatches, and skips, optimizing for subsequence alignment under temporal distortions.
• A frame-level attribution model classifies each audio frame as matching or non-matching using bidirectional LSTM and Transformer architectures.
• The system is trained on synthetic tampering scenarios involving insertions, deletions, and replacements of varying durations (0.1s to 5s).
• The framework is evaluated on tampered recordings of Donald Trump, using audio from multiple sources and varying bitrates (96kbps to 160kbps).

Experimental results

Research questions

• RQ1 Can a cross-verification system based on trusted reference recordings reliably detect tampered audio from world leaders, even when the tampering is indistinguishable from genuine content?
• RQ2 How does a modified Needleman-Wunsch algorithm (SNWTW) compare to dynamic time warping in aligning tampered audio sequences with reference recordings?
• RQ3 What is the performance of deep learning models (LSTM, BiLSTM, Transformer) in classifying frame-level matches between query and reference audio?
• RQ4 How do edit duration and audio bitrate affect the system’s alignment and classification accuracy?
• RQ5 Can the system maintain high accuracy across diverse tampering operations and real-world audio quality variations?

Key findings

• The SNWTW alignment method achieves 99.7% accuracy at a 50 ms error tolerance, significantly outperforming dynamic time warping, which stabilizes at around 2.8% error rate.
• The best-performing model, a 3-layer bidirectional LSTM, achieves a 0.43% equal error rate in frame-level attribution, outperforming both standard LSTM and Transformer models.
• Performance degrades with increasing edit duration deviation from the training duration (L=2s), with EER rising to 15.14% for 0.1s edits, but remains strong for 1s and 2s edits.
• Audio bitrate has negligible impact on performance, with stable results even at 96kbps, indicating robustness across quality levels.
• The system maintains high accuracy across various tampering operations, including insertions, deletions, and replacements, demonstrating generalization to realistic attack scenarios.
• The cross-verification approach enables positive verification of genuine recordings, offering a permanent solution independent of the sophistication of deepfake technology.