[Paper Review] Accenture at CheckThat! 2021: Interesting claim identification and ranking with contextually sensitive lexical training data augmentation

Evan R. Williams, Paul Rodrigues+1 more|arXiv (Cornell University)|Jul 12, 2021

Topic Modeling4 citations

TL;DR

This paper presents a contextually sensitive lexical data augmentation method using BERT and RoBERTa to improve claim identification and ranking for fact-checking in multilingual social media. By replacing key words with semantically appropriate synonyms based on contextual embeddings, the approach significantly boosted performance—achieving the highest mAP (0.658) in the Arabic track at CLEF 2021 CheckThat! Lab, outperforming other languages due to effective data synthesis and model sensitivity to training data quantity.

ABSTRACT

This paper discusses the approach used by the Accenture Team for CLEF2021 CheckThat! Lab, Task 1, to identify whether a claim made in social media would be interesting to a wide audience and should be fact-checked. Twitter training and test data were provided in English, Arabic, Spanish, Turkish, and Bulgarian. Claims were to be classified (check-worthy/not check-worthy) and ranked in priority order for the fact-checker. Our method used deep neural network transformer models with contextually sensitive lexical augmentation applied on the supplied training datasets to create additional training samples. This augmentation approach improved the performance for all languages. Overall, our architecture and data augmentation pipeline produced the best submitted system for Arabic, and performance scales according to the quantity of provided training data for English, Spanish, Turkish, and Bulgarian. This paper investigates the deep neural network architectures for each language as well as the provided data to examine why the approach worked so effectively for Arabic, and discusses additional data augmentation measures that should could be useful to this problem.

Motivation & Objective

• To improve the identification and ranking of check-worthy claims in multilingual social media for automated fact-checking.
• To address the challenge of imbalanced training data, especially for low-resource languages like Arabic, Bulgarian, Turkish, and Spanish.
• To enhance model generalization and performance through contextually informed data augmentation without relying on external data sources.
• To investigate whether lexical augmentation using contextual embeddings can outperform random or non-contextual augmentation in low-resource NLP settings.
• To determine the optimal augmentation intensity (probability) for balancing performance gains and computational cost across multiple languages.

Proposed method

• The method employs BERT and RoBERTa models to generate contextually appropriate synonym replacements for words in check-worthy claims from the training set.
• For each claim, the model computes token-level embedding probabilities and selects replacements with the highest likelihood under the contextual distribution.
• A hyperparameter p controls the probability of replacing tokens; p = 0.1 was selected after ablation studies to balance performance and computational cost.
• Augmented samples were appended to the original training data with the same 'check-worthy' label, increasing the size of the positive class.
• The final model architecture includes a mean-pooling layer and dropout before the classification head to prevent overfitting during fine-tuning.
• Ranking was performed using the softmax output of the model, with the difference between positive and negative class probabilities used to prioritize claims.

Experimental results

Research questions

• RQ1 Does contextually sensitive lexical data augmentation improve claim detection performance across multiple low-resource languages in fact-checking tasks?
• RQ2 How does the performance of the model vary with different levels of data augmentation intensity (p) across languages?
• RQ3 Why did the method achieve superior results in Arabic despite similar augmentation application across all languages?
• RQ4 Can contextual embedding-based augmentation outperform random or non-contextual augmentation in low-resource NLP scenarios?
• RQ5 How does the quantity and distribution of training data interact with data augmentation to influence model performance?

Key findings

• The contextually sensitive lexical augmentation method improved the F1 score for the 'check-worthy' class across all five languages (Arabic, Bulgarian, English, Spanish, Turkish) on the development set.
• The system achieved the highest mean average precision (mAP) of 0.658 in the Arabic track, outperforming all other submissions in the competition.
• Performance scaled positively with training data size, with higher mAP values observed in languages that received more training samples (e.g., Turkish and Arabic with 3,095 samples each).
• The model achieved an mAP of 0.497 in Bulgarian and 0.491 in Spanish, indicating strong performance in medium-resource languages.
• The method showed a performance drop in English (mAP = 0.101), suggesting that the approach may be less effective when baseline data is already relatively abundant.
• Ablation studies confirmed that p = 0.1 provided the best trade-off between performance gain and computational cost, with higher values leading to diminishing returns.