[Paper Review] AI model GPT-3 (dis)informs us better than humans

Giovanni Spitale, Nikola Biller‐Andorno+1 more|Zurich Open Repository and Archive (University of Zurich)|Jan 23, 2023

Misinformation and Its ImpactsSocial Sciences15 references10 citations

TL;DR

The study shows GPT-3 can generate accurate, easy-to-understand information but can also create more compelling disinformation, and humans often cannot tell GPT-3 tweets from human ones.

ABSTRACT

Artificial intelligence is changing the way we create and evaluate information, and this is happening during an infodemic, which has been having dramatic effects on global health. In this paper we evaluate whether recruited individuals can distinguish disinformation from accurate information, structured in the form of tweets, and determine whether a tweet is organic or synthetic, i.e., whether it has been written by a Twitter user or by the AI model GPT-3. Our results show that GPT-3 is a double-edge sword, which, in comparison with humans, can produce accurate information that is easier to understand, but can also produce more compelling disinformation. We also show that humans cannot distinguish tweets generated by GPT-3 from tweets written by human users. Starting from our results, we reflect on the dangers of AI for disinformation, and on how we can improve information campaigns to benefit global health.

Motivation & Objective

• Assess whether recruited individuals can distinguish disinformation from accurate information in tweet form.
• Evaluate whether tweets are organic (human-written) or synthetic (GPT-3 written).
• Explore how GPT-3 compares to humans in producing information that is accurate and easy to understand.
• Discuss implications for AI-driven disinformation and strategies to improve health information campaigns.

Proposed method

• Present tweets structured as disinformation and accurate information to participants.
• Compare accuracy and understandability of information produced by GPT-3 versus humans.
• Ask participants to classify tweets as organic or synthetic.
• Analyze participants’ ability to distinguish GPT-3 content from human content.
• Leverage dataset and software resources referenced by the authors.

Experimental results

Research questions

• RQ1 Can participants reliably distinguish disinformation from accurate information in tweet form?
• RQ2 Can participants distinguish tweets written by GPT-3 from those written by humans (organic vs synthetic)?
• RQ3 How does GPT-3’s information quality and comprehensibility compare to human-generated content?
• RQ4 What are the implications of AI-generated content for health information campaigns?

Key findings

• GPT-3 can produce accurate information that is easier to understand than some human-generated content.
• GPT-3 can also generate more compelling disinformation than humans.
• Humans struggle to distinguish tweets generated by GPT-3 from those written by human users.
• The results raise concerns about AI-driven disinformation and emphasize the need to improve information campaigns for global health.
• The paper discusses dangers of AI for disinformation and potential strategies to mitigate them.