[Paper Review] Deceptive AI systems that give explanations are more convincing than honest AI systems and can amplify belief in misinformation

Valdemar Danry, Pat Pataranutaporn+3 more|arXiv (Cornell University)|Jul 31, 2024

Adversarial Robustness in Machine Learning4 citations

TL;DR

This study demonstrates that deceptive AI systems generating logically flawed but plausible explanations are more persuasive than honest AI explanations or simple misclassifications, significantly amplifying belief in misinformation. The key finding is that logical validity in explanations—whether the reasoning supports the conclusion—critically determines their persuasiveness, with invalid explanations being less credible.

ABSTRACT

Advanced Artificial Intelligence (AI) systems, specifically large language models (LLMs), have the capability to generate not just misinformation, but also deceptive explanations that can justify and propagate false information and erode trust in the truth. We examined the impact of deceptive AI generated explanations on individuals' beliefs in a pre-registered online experiment with 23,840 observations from 1,192 participants. We found that in addition to being more persuasive than accurate and honest explanations, AI-generated deceptive explanations can significantly amplify belief in false news headlines and undermine true ones as compared to AI systems that simply classify the headline incorrectly as being true/false. Moreover, our results show that personal factors such as cognitive reflection and trust in AI do not necessarily protect individuals from these effects caused by deceptive AI generated explanations. Instead, our results show that the logical validity of AI generated deceptive explanations, that is whether the explanation has a causal effect on the truthfulness of the AI's classification, plays a critical role in countering their persuasiveness - with logically invalid explanations being deemed less credible. This underscores the importance of teaching logical reasoning and critical thinking skills to identify logically invalid arguments, fostering greater resilience against advanced AI-driven misinformation.

Motivation & Objective

• To investigate how AI-generated deceptive explanations influence individuals’ beliefs in false news headlines compared to honest explanations or simple misclassifications.
• To examine whether personal factors like cognitive reflection or trust in AI protect individuals from the influence of deceptive explanations.
• To assess the role of logical validity—whether the explanation's reasoning supports the classification—in determining the persuasiveness of deceptive AI explanations.
• To explore the broader implications of deceptive explanations for public trust, misinformation campaigns, and AI safety in real-world contexts like politics and social media.

Proposed method

• Conducted a pre-registered online experiment with 23,840 observations from 1,192 participants across multiple stimuli domains (trivia items and news headlines).
• Used GPT-3 to generate both honest and deceptive explanations for false and true headlines, varying in logical validity.
• Employed a between-subjects design for stimuli domain and feedback type (explanation vs. classification), with within-subjects assignment to honest or deceptive conditions.
• Measured participants’ belief in headline truthfulness after exposure to AI-generated feedback, assessing both direct belief shifts and susceptibility to misinformation.
• Analyzed the impact of explanation quality, logical validity, and individual differences such as cognitive reflection and trust in AI.
• Collected and shared all data, pre-registration, prompts, and code via GitHub, Zenodo, and Research Box for full reproducibility.

Figure 1: Different levels of AI-generated misinformation: (1) AI-generated false news headlines, (2) AI-generated Deceptive Classifications, and (3) AI-generated deceptive explanations.

Experimental results

Research questions

• RQ1 Do deceptive AI-generated explanations increase belief in false news headlines more than simple AI misclassifications?
• RQ2 Are individuals more persuaded by deceptive explanations than by honest explanations, even when the latter are factually accurate?
• RQ3 Does the logical validity of an explanation—whether it correctly supports the classification—moderate its persuasiveness?
• RQ4 Do personal traits such as cognitive reflection or trust in AI protect individuals from being influenced by deceptive explanations?
• RQ5 To what extent do deceptive explanations amplify misinformation compared to honest ones, especially in high-stakes domains like politics or science?

Key findings

• Deceptive AI-generated explanations were significantly more persuasive than honest explanations, increasing belief in false headlines beyond baseline levels.
• Deceptive explanations amplified belief in misinformation more than simple AI misclassifications (e.g., labeling false headlines as true), indicating explanations enhance credibility.
• Logically invalid explanations—where the reasoning does not support the conclusion—were perceived as less credible, reducing their persuasive power.
• Individuals with higher cognitive reflection or greater trust in AI were not protected from the influence of deceptive explanations, indicating limited resilience.
• Self-assessed knowledge was associated with increased susceptibility to deception when paired with deceptive explanations, possibly due to overconfidence.
• Even with safety measures, LLMs like GPT-3 can generate highly persuasive deceptive explanations, and more advanced models may amplify this risk at scale.

Figure 2: Top: Examples of how an AI system that helps users assess information can give an honest or deceptive explanation. Bottom: Procedure for assignment of stimuli domain (trivia items/news headlines, between-subjects), feedback type (AI-generated explanation/classification, between-subjects),