[Paper Review] Identifying Misinformation on YouTube through Transcript Contextual Analysis with Transformer Models

Christos Christodoulou, Nikos Salamanos+3 more|arXiv (Cornell University)|Jul 22, 2023

Misinformation and Its ImpactsSocial Sciences3 citations

TL;DR

This paper proposes a text-based misinformation detection system for YouTube videos using fine-tuned and few-shot fine-tuned transformer models (e.g., RoBERTa, ELECTRA, MPNet) on video transcripts. It achieves high performance—F1 > 0.90 and MCC > 0.81—on vaccine and fake-news datasets, with few-shot models outperforming fine-tuned models by 20% on the YouTube pseudoscience dataset, demonstrating strong generalization in low-data regimes.

ABSTRACT

Misinformation on YouTube is a significant concern, necessitating robust detection strategies. In this paper, we introduce a novel methodology for video classification, focusing on the veracity of the content. We convert the conventional video classification task into a text classification task by leveraging the textual content derived from the video transcripts. We employ advanced machine learning techniques like transfer learning to solve the classification challenge. Our approach incorporates two forms of transfer learning: (a) fine-tuning base transformer models such as BERT, RoBERTa, and ELECTRA, and (b) few-shot learning using sentence-transformers MPNet and RoBERTa-large. We apply the trained models to three datasets: (a) YouTube Vaccine-misinformation related videos, (b) YouTube Pseudoscience videos, and (c) Fake-News dataset (a collection of articles). Including the Fake-News dataset extended the evaluation of our approach beyond YouTube videos. Using these datasets, we evaluated the models distinguishing valid information from misinformation. The fine-tuned models yielded Matthews Correlation Coefficient>0.81, accuracy>0.90, and F1 score>0.90 in two of three datasets. Interestingly, the few-shot models outperformed the fine-tuned ones by 20% in both Accuracy and F1 score for the YouTube Pseudoscience dataset, highlighting the potential utility of this approach -- especially in the context of limited training data.

Motivation & Objective

• To develop a robust, scalable method for detecting misinformation on YouTube using only video transcripts.
• To evaluate the effectiveness of fine-tuned transformer models (e.g., BERT, RoBERTa, ELECTRA) and few-shot learning (e.g., MPNet, RoBERTa-large) in classifying misinformation.
• To address the challenge of long document classification in transcripts using overlapping windowing strategies.
• To assess model performance across diverse misinformation types: vaccine misinformation, pseudoscience, and fake news articles.
• To provide open-source code for reproducibility and broader adoption in misinformation detection research.

Proposed method

• Converts video classification into text classification by extracting and analyzing textual transcripts from YouTube videos.
• Applies transfer learning via fine-tuning pre-trained transformer models (BERT, RoBERTa, ELECTRA) on task-specific datasets.
• Employs few-shot learning using sentence-transformers (MPNet, RoBERTa-large) to enable high performance with minimal labeled data.
• Uses a sliding window strategy with 80% overlap to handle long transcripts, ensuring no critical information is lost.
• Applies the formula $ N = \lceil \frac{L - M}{S} \rceil + 1 $ to determine the number of overlapping windows, where $ L $ is document length, $ M $ is max sequence length, and $ S $ is overlap.
• Classifies each transcript window and aggregates predictions to determine the final label (misinformation or valid content).

Experimental results

Research questions

• RQ1 RQ1: How effective are fine-tuned transformer models and few-shot learning in detecting misinformation in YouTube videos using only the video transcripts?
• RQ2 RQ2: How does the performance of these models vary across different types of datasets?
• RQ3 RQ3: How can we effectively handle long document classification using transformer models?
• RQ4 RQ4: What is the comparative performance of fine-tuned versus few-shot models in low-data regimes?

Key findings

• Fine-tuned RoBERTa achieved an MCC of 0.88, accuracy of 0.94, and F1 score of 0.94 on the YouTube Vaccine dataset.
• On the YouTube Pseudoscience dataset, few-shot MPNet outperformed all fine-tuned models, achieving an F1 score of 0.78 and accuracy of 0.72, surpassing fine-tuned models by 20% in both metrics.
• Fine-tuned ELECTRA achieved the highest performance on the ISOT Fake News dataset, with an MCC of 0.94, accuracy of 0.97, and F1 score of 0.97.
• Few-shot RoBERTa-large achieved an F1 score of 0.90 and accuracy of 0.94 on the ISOT Fake News dataset, outperforming MPNet in few-shot settings.
• The sliding window strategy with 80% overlap effectively preserved long-context information, enabling reliable classification of lengthy transcripts.
• The study demonstrates that few-shot learning is particularly effective in low-resource settings, such as the YouTube Pseudoscience dataset, where labeled data is scarce.