[Paper Review] On the Benefit of Combining Neural, Statistical and External Features for Fake News Identification

Gaurav Bhatt, Aman Sharma+4 more|arXiv (Cornell University)|Dec 11, 2017

Misinformation and Its Impacts21 references18 citations

TL;DR

This paper proposes a hybrid model that integrates skip-thought sentence embeddings, TF-IDF weighted n-gram features, and hand-crafted external features to improve fake news stance detection. By combining these neural, statistical, and heuristic features through a deep neural network, the approach achieves state-of-the-art performance on the FNC-1 dataset, with an overall FNC score of 89.29 and superior accuracy on the 'discuss' stance (85.68%) and 'unrelated' stance (98.04%).

ABSTRACT

Identifying the veracity of a news article is an interesting problem while automating this process can be a challenging task. Detection of a news article as fake is still an open question as it is contingent on many factors which the current state-of-the-art models fail to incorporate. In this paper, we explore a subtask to fake news identification, and that is stance detection. Given a news article, the task is to determine the relevance of the body and its claim. We present a novel idea that combines the neural, statistical and external features to provide an efficient solution to this problem. We compute the neural embedding from the deep recurrent model, statistical features from the weighted n-gram bag-of-words model and handcrafted external features with the help of feature engineering heuristics. Finally, using deep neural layer all the features are combined, thereby classifying the headline-body news pair as agree, disagree, discuss, or unrelated. We compare our proposed technique with the current state-of-the-art models on the fake news challenge dataset. Through extensive experiments, we find that the proposed model outperforms all the state-of-the-art techniques including the submissions to the fake news challenge.

Motivation & Objective

• To address the challenge of fake news identification by focusing on stance detection between headlines and news bodies.
• To investigate whether combining neural, statistical, and external features improves performance on fake news stance classification.
• To overcome the limitations of deep learning models on imbalanced datasets, particularly for the 'disagree' stance category.
• To develop a robust, hybrid model that leverages multiple feature types for improved generalization and accuracy.
• To outperform existing state-of-the-art models on the FNC-1 benchmark dataset using a multi-modal feature integration strategy.

Proposed method

• Utilizes skip-throw vectors to generate dense, contextual sentence embeddings for both headlines and news bodies.
• Applies TF-IDF weighted n-gram bag-of-words representation to extract statistical features from textual content.
• Incorporates hand-crafted external features such as word overlap, cosine similarity, and n-gram matching between headline and body.
• Combines all feature types—neural, statistical, and external—into a unified deep neural network for end-to-end classification.
• Employs a deep feedforward network to classify stance into four categories: agree, disagree, discuss, or unrelated.
• Trains and evaluates the model on the FNC-1 dataset, using the official FNC score and class-wise accuracy as evaluation metrics.

Experimental results

Research questions

• RQ1 Can the integration of neural, statistical, and external features improve stance detection performance on fake news datasets?
• RQ2 How does the proposed hybrid model compare to state-of-the-art deep learning architectures on the FNC-1 benchmark?
• RQ3 Why do existing models underperform on the 'disagree' stance, and can feature engineering help mitigate this?
• RQ4 Does combining skip-though embeddings with TF-IDF and heuristic features lead to better generalization on imbalanced data?
• RQ5 To what extent do external features contribute to improved performance on the 'unrelated' and 'discuss' stances?

Key findings

• The proposed model achieves an overall FNC score of 89.29, outperforming all state-of-the-art models, including top submissions to the FNC-1 challenge.
• The model achieves the highest class-wise accuracy on the 'discuss' stance (85.68%), significantly outperforming other models.
• It attains 98.04% accuracy on the 'unrelated' stance, demonstrating strong generalization on this challenging category.
• Despite a slightly lower overall accuracy than one top submission, the model's FNC score is higher, indicating better balance across all stances.
• The confusion matrix shows strong performance on 'agree' (43.82%) and 'disagree' (6.31%), with the latter being particularly difficult for most models.
• The model's success is attributed to the effective fusion of skip-though embeddings, TF-IDF n-grams, and hand-crafted features, which compensates for data imbalance and model bias.