[Paper Review] Team Yao at Factify 2022: Utilizing Pre-trained Models and Co-attention Networks for Multi-Modal Fact Verification

Wei‐Yao Wang, Wen-Chih Peng|arXiv (Cornell University)|Jan 26, 2022

Misinformation and Its Impacts4 citations

TL;DR

This paper proposes Pre-CoFact, a multi-modal fact verification framework using pre-trained DeBERTa and DeiT models for text and image feature extraction, combined with co-attention networks to fuse cross-modal and same-modality representations. The method achieved a 74.585% F1-score, securing 5th place in the Factify 2022 challenge without using auxiliary tasks or OCR data.

ABSTRACT

In recent years, social media has enabled users to get exposed to a myriad of misinformation and disinformation; thus, misinformation has attracted a great deal of attention in research fields and as a social issue. To address the problem, we propose a framework, Pre-CoFact, composed of two pre-trained models for extracting features from text and images, and multiple co-attention networks for fusing the same modality but different sources and different modalities. Besides, we adopt the ensemble method by using different pre-trained models in Pre-CoFact to achieve better performance. We further illustrate the effectiveness from the ablation study and examine different pre-trained models for comparison. Our team, Yao, won the fifth prize (F1-score: 74.585\%) in the Factify challenge hosted by De-Factify @ AAAI 2022, which demonstrates that our model achieved competitive performance without using auxiliary tasks or extra information. The source code of our work is publicly available at https://github.com/wywyWang/Multi-Modal-Fact-Verification-2021

Motivation & Objective

• To address the growing challenge of misinformation spread through social media, particularly multi-modal fake news combining text and images.
• To develop a robust fact verification system that leverages both textual and visual content without relying on OCR or external data.
• To improve performance in multi-modal fact verification by modeling cross-modal and intra-modal dependencies through co-attention mechanisms.
• To evaluate the effectiveness of different pre-trained models and ensemble strategies in a zero-shot or minimal-data setting.

Proposed method

• The framework uses DeBERTa-base for encoding textual claims and documents, and DeiT-base-patch16-224 for encoding images.
• Co-attention modules are applied to fuse features across modalities (text and image) and within the same modality (claim and document text/images).
• Four embedding representations—text, image, cross-attention, and co-attention—are concatenated and classified into one of five categories.
• An ensemble strategy combines multiple variants of Pre-CoFact using different pre-trained models (e.g., RoBERTa, XLM-RoBERTa) and activation functions (e.g., Mish) to improve generalization.
• Model parameters for DeBERTa and DeiT are frozen during fine-tuning to preserve pre-trained knowledge.
• Data preprocessing includes image resizing to 256x256, center cropping to 224x224, and normalization; text sequences are truncated to 512 tokens.

Experimental results

Research questions

• RQ1 Can a multi-modal fact verification system achieve competitive performance using only raw text and images, without OCR or auxiliary data?
• RQ2 How effective are co-attention mechanisms in modeling cross-modal and intra-modal dependencies for fact verification?
• RQ3 Which pre-trained models (e.g., DeBERTa, RoBERTa, XLM-RoBERTa, DeiT, DINO) yield the best performance in this multi-modal setting?
• RQ4 To what extent does model ensembling improve robustness and performance in multi-modal fact verification?

Key findings

• The proposed Pre-CoFact framework achieved a weighted F1-score of 78.46% on the validation set, with an ensemble model reaching 80.02%.
• The ablation study confirmed that co-attention mechanisms significantly improve performance, with a 4.34% drop in F1 when co-attention is removed.
• Using co-attention only within the same modality (e.g., text-text or image-image) yielded lower performance (76.43%) than the full co-attention model (78.46%), indicating the necessity of cross-modal attention.
• The ensemble method improved performance by 1.56 percentage points over the base Pre-CoFact model, demonstrating its effectiveness in boosting generalization.
• Among pre-trained models, DeBERTa and DeiT outperformed XLM-RoBERTa and RoBERTa, with DeiT showing better performance than DINO on this task.
• The model achieved a test F1-score of 74.585%, securing 5th place in the Factify 2022 challenge, outperforming the baseline by 40.5%.