[Paper Review] Veracity Computing from Lexical Cues and Perceived Certainty Trends

Uwe D. Reichel, Piroska Lendvai|arXiv (Cornell University)|Nov 8, 2016

Misinformation and Its Impacts24 references4 citations

TL;DR

This paper proposes a data-driven method for veracity computation in social media rumors using lexical cues and predicted certainty trends. It combines lexical cue ratios and time-series patterns of certainty to identify rumor-resolving tweets and predict resolution truth value, achieving 0.74 F1-score for resolution detection and 0.76 F1-score for truth value prediction without external resources.

ABSTRACT

We present a data-driven method for determining the veracity of a set of rumorous claims on social media data. Tweets from different sources pertaining to a rumor are processed on three levels: first, factuality values are assigned to each tweet based on four textual cue categories relevant for our journalism use case; these amalgamate speaker support in terms of polarity and commitment in terms of certainty and speculation. Next, the proportions of these lexical cues are utilized as predictors for tweet certainty in a generalized linear regression model. Subsequently, lexical cue proportions, predicted certainty, as well as their time course characteristics are used to compute veracity for each rumor in terms of the identity of the rumor-resolving tweet and its binary resolution value judgment. The system operates without access to extralinguistic resources. Evaluated on the data portion for which hand-labeled examples were available, it achieves .74 F1-score on identifying rumor resolving tweets and .76 F1-score on predicting if a rumor is resolved as true or false.

Motivation & Objective

• To automatically determine the veracity of rumors in social media using only textual cues and temporal dynamics.
• To identify which tweet in a rumor thread resolves the claim, based on linguistic and temporal features.
• To predict whether a rumor is ultimately resolved as true or false using only in-text and time-anchored features.
• To model speaker certainty from lexical cues without relying on external metadata or source profiles.
• To evaluate the impact of temporal shifts in cue ratios and certainty on claim resolution detection.

Proposed method

• Four lexical cue categories—knowledge, report, belief, and doubt—are used to assign factuality values to tweets based on polarity and commitment.
• Generalized linear models (GLMs) predict tweet certainty from ratios of these lexical cues, addressing weak individual cue correlations.
• Time-series analysis identifies discontinuities (e.g., resets, deltas) in cue ratios and predicted certainty to detect resolution points.
• Feature sets combining cue ratios (CueSet) and predicted certainty (CertSet) are used as input for AdaBoostM1 classifiers.
• 10-fold cross-validation with resampling is applied to handle class imbalance in the held-out test sets.
• The system operates without access to extralinguistic data, relying solely on textual and temporal patterns.

Experimental results

Research questions

• RQ1 Can lexical cue ratios and predicted certainty levels reliably identify the tweet that resolves a rumor?
• RQ2 How do temporal trends in cue ratios and certainty values differ between verified and falsified claims?
• RQ3 To what extent do discontinuities in cue ratios and certainty scores signal claim resolution?
• RQ4 Does integrating predicted certainty improve resolution value prediction compared to raw cue ratios?
• RQ5 How do belief and doubt cues relate to the final resolution outcome of a rumor?

Key findings

• The system achieved a 0.74 F1-score in identifying the correct rumor-resolving tweet using lexical cue ratios (CueSet).
• For predicting whether a rumor is true or false, the model reached a 0.76 F1-score using predicted certainty values (CertSet).
• Resolving tweets show significant increases in knowledge and report cue ratios (KCR, RCR), with a peak in factuality cue ratio (FCR) at resolution time.
• Doubt cues increase more sharply after resolution in falsified claims, while certainty cues rise in verified claims and drop in falsified ones.
• The interaction between resolution value and certainty trends shows that direction and magnitude of discontinuities depend on whether the claim is ultimately true or false.
• Predicted certainty features outperform raw cue ratios for resolution value prediction, indicating benefit from intermediate feature integration in sparse data.