[論文レビュー] A simple but tough-to-beat baseline for the Fake News Challenge stance detection task

Benjamin Riedel, Isabelle Augenstein+2 more|arXiv (Cornell University)|Jul 11, 2017

Topic Modeling参考文献 8被引用数 206

ひとこと要約

この論文は、FNC-1のためのシンプルなエンドツーエンドのスタンス検出システムを提案し、bag-of-words特徴と小さなMLPを使用して競争力のある結果を出し、Fake News ChallengeのStage 1で3位に入賞した。著者らはそのシンプルさにもかかわらず強力なベースラインとして推奨する。

ABSTRACT

Identifying public misinformation is a complicated and challenging task. An important part of checking the veracity of a specific claim is to evaluate the stance different news sources take towards the assertion. Automatic stance evaluation, i.e. stance detection, would arguably facilitate the process of fact checking. In this paper, we present our stance detection system which claimed third place in Stage 1 of the Fake News Challenge. Despite our straightforward approach, our system performs at a competitive level with the complex ensembles of the top two winning teams. We therefore propose our system as the 'simple but tough-to-beat baseline' for the Fake News Challenge stance detection task.

研究の動機と目的

• Motivate the utility of a simple stance-detection baseline for FNC-1 to aid fact-checking workflows.
• Develop an end-to-end system using lightweight linguistic features to evaluate stance between headlines and bodies.
• Evaluate the system against more complex ensembles and establish its standing as a baseline.
• Provide reproducible implementation and analysis to inform future baselines in stance detection.

提案手法

• Two bag-of-words representations (TF and TF-IDF) are extracted for headline and body.
• A cosine similarity is computed between L2-normalized TF-IDF vectors of headline and body.
• The TF and TF-IDF features are concatenated into a 10,001-dimensional feature vector and fed to a one-hidden-layer MLP with ReLU activations.
• Training uses cross-entropy loss with L2 regularization and dropout, optimized with Adam and gradient clipping.
• Hyperparameters are tuned via random search with cross-validation on data splits.

実験結果

リサーチクエスチョン

• RQ1 Can a simple BOW-based representation with a small neural classifier compete with more complex ensembles in FNC-1 stance detection?
• RQ2 What is the performance of a lightweight baseline on the key labels of interest (agree, disagree) compared to top systems?
• RQ3 How do simple features like TF, TF-IDF, and their cosine similarity contribute to stance classification?

主な発見

• The system achieved an FNC-1 score of 81.72% on the final test set, placing third among 50 teams.
• High accuracy (96.55%) is achieved on distinguishing related vs. unrelated pairs, with most misclassifications occurring in the more nuanced agree/disagree dimensions.
• The approach is competitive with top ensemble methods despite its simplicity.
• The authors provide open-source code and claim reproducibility via a public GitHub repository.
• The baseline outperformed many participants and closely trailed the top two teams.
• The confusion matrix reveals average performance for 'agree' but weaker performance for 'disagree'.