[Paper Review] A personal model of trumpery: Deception detection in a real-world high-stakes setting

Sophie van der Zee, Ronald Poppe+2 more|arXiv (Cornell University)|Nov 5, 2018

Deception detection and forensic psychologyPsychology14 references3 citations

TL;DR

This study develops a personalized linguistic model to detect deception in real-world high-stakes communication by analyzing tweets from the U.S. president, using fact-checks from the Washington Post as ground truth. It achieves 73% accuracy in predicting factual correctness of out-of-sample tweets, demonstrating that deceptive language patterns are systematic and identifiable at the individual level.

ABSTRACT

Language use reveals information about who we are and how we feel1-3. One of the pioneers in text analysis, Walter Weintraub, manually counted which types of words people used in medical interviews and showed that the frequency of first-person singular pronouns (i.e., I, me, my) was a reliable indicator of depression, with depressed people using I more often than people who are not depressed4. Several studies have demonstrated that language use also differs between truthful and deceptive statements5-7, but not all differences are consistent across people and contexts, making prediction difficult8. Here we show how well linguistic deception detection performs at the individual level by developing a model tailored to a single individual: the current US president. Using tweets fact-checked by an independent third party (Washington Post), we found substantial linguistic differences between factually correct and incorrect tweets and developed a quantitative model based on these differences. Next, we predicted whether out-of-sample tweets were either factually correct or incorrect and achieved a 73% overall accuracy. Our results demonstrate the power of linguistic analysis in real-world deception research when applied at the individual level and provide evidence that factually incorrect tweets are not random mistakes of the sender.

Motivation & Objective

• To investigate whether linguistic patterns in social media posts can reliably predict factual accuracy at the individual level.
• To develop a personalized model of deception detection tailored to a single high-profile individual, specifically the U.S. president.
• To assess whether factually incorrect tweets are random errors or reflect systematic linguistic patterns.
• To evaluate the performance of a quantitative linguistic model in predicting factual correctness in real-world, high-stakes communication.

Proposed method

• Linguistic features were extracted from 1,000 tweets by the U.S. president, labeled as factually correct or incorrect by the Washington Post's fact-checking team.
• A personal model was trained on these labeled tweets using supervised machine learning to identify linguistic markers of deception.
• The model focused on lexical, syntactic, and pragmatic features, including pronoun use, sentiment, and structural complexity.
• The model was tested on out-of-sample tweets not used in training to evaluate generalization performance.
• Prediction accuracy was measured using standard classification metrics, with 73% overall accuracy reported.

Experimental results

Research questions

• RQ1 Can a personalized linguistic model detect deception in high-stakes social media communication with high accuracy?
• RQ2 Are there consistent linguistic differences between factually correct and incorrect tweets from a single individual?
• RQ3 Do factually incorrect tweets from the same individual exhibit systematic linguistic patterns rather than random errors?
• RQ4 How well does a model trained on one individual generalize to unseen tweets from the same source?

Key findings

• The personalized linguistic model achieved 73% accuracy in predicting whether a tweet was factually correct or incorrect.
• Substantial linguistic differences were found between factually correct and incorrect tweets from the U.S. president.
• The model demonstrated that deceptive tweets are not random mistakes but reflect consistent patterns in language use.
• The results support the feasibility of individual-level deception detection using linguistic analysis in real-world, high-stakes settings.