[Paper Review] Labeling Messages as AI-Generated Does Not Reduce Their Persuasive Effects

Isabel O. Gallegos, Chen Shani+5 more|ArXiv.org|Apr 14, 2025

Psychology of Moral and Emotional Judgment3 citations

TL;DR

The study finds that labeling AI-generated messages as AI, human, or unlabeled does not significantly affect their persuasiveness across four policy domains.

ABSTRACT

As generative artificial intelligence (AI) enables the creation and dissemination of information at massive scale and speed, it is increasingly important to understand how people perceive AI-generated content. One prominent policy proposal requires explicitly labeling AI-generated content to increase transparency and encourage critical thinking about the information, but prior research has not yet tested the effects of such labels. To address this gap, we conducted a survey experiment (N=1601) on a diverse sample of Americans, presenting participants with an AI-generated message about several public policies (e.g., allowing colleges to pay student-athletes), randomly assigning whether participants were told the message was generated by (a) an expert AI model, (b) a human policy expert, or (c) no label. We found that messages were generally persuasive, influencing participants' views of the policies by 9.74 percentage points on average. However, while 94.6% of participants assigned to the AI and human label conditions believed the authorship labels, labels had no significant effects on participants' attitude change toward the policies, judgments of message accuracy, nor intentions to share the message with others. These patterns were robust across a variety of participant characteristics, including prior knowledge of the policy, prior experience with AI, political party, education level, or age. Taken together, these results imply that, while authorship labels would likely enhance transparency, they are unlikely to substantially affect the persuasiveness of the labeled content, highlighting the need for alternative strategies to address challenges posed by AI-generated information.

Motivation & Objective

• Assess whether AI authorship labels alter the persuasiveness of AI-generated policy messages.
• Test three labeling conditions (AI-generated, human-written, and no label) across multiple policy topics.
• Measure changes in policy support and related perceptions before and after exposure to labeled messages.
• Evaluate robustness across participant characteristics such as prior knowledge, AI experience, and demographics.

Proposed method

• Pre-registered survey experiment with N=1,601 participants from a diverse American sample.
• Random assignment to AI label, human label, or no label conditions for four policy proposals.
• Messages generated by GPT-4o and edited for factual accuracy using evidence-based persuasion techniques.
• Outcome measures include change in policy support (primary), confidence, accuracy judgments, and sharing intentions on 0–100 scales.
• Regression analyses controlling for policy and pre-intervention support to compare labeling conditions.

Experimental results

Research questions

• RQ1 Does labeling AI-generated content as AI, human, or no label affect its persuasiveness on policy attitudes?
• RQ2 Are there differences in perceived message accuracy, confidence in support, or sharing intentions across labeling conditions?
• RQ3 Do participants’ characteristics (prior knowledge, AI experience, political identity, education, age) moderate labeling effects?
• RQ4 Is there robust consistency of labeling effects across multiple policy domains?

Key findings

• Messages were persuasive overall, changing policy support by 9.74 percentage points on average.
• 94.6% of AI-labeled and 89.3% of human-labeled participants believed the assigned authorship label.
• There were no significant differences in persuasiveness between AI-labeled, human-labeled, and no-label conditions.
• No significant differences emerged for confidence in support, message accuracy judgments, or sharing intentions across labeling conditions.