[Paper Review] Disinformation Capabilities of Large Language Models

Ivan Vykopal, Matúš Pikuliak+4 more|arXiv (Cornell University)|Nov 15, 2023

Misinformation and Its Impacts4 citations

TL;DR

This paper evaluates the disinformation capabilities of 10 instruction-tuned large language models (LLMs) in generating convincing English news articles aligned with 20 harmful disinformation narratives, such as health hoaxes. It finds that LLMs, including open-source models, can produce highly plausible disinformation with minimal safety filter failure, are detectable by existing models, and can be partially automated for evaluation using LLMs themselves.

ABSTRACT

Automated disinformation generation is often listed as an important risk associated with large language models (LLMs). The theoretical ability to flood the information space with disinformation content might have dramatic consequences for societies around the world. This paper presents a comprehensive study of the disinformation capabilities of the current generation of LLMs to generate false news articles in the English language. In our study, we evaluated the capabilities of 10 LLMs using 20 disinformation narratives. We evaluated several aspects of the LLMs: how good they are at generating news articles, how strongly they tend to agree or disagree with the disinformation narratives, how often they generate safety warnings, etc. We also evaluated the abilities of detection models to detect these articles as LLM-generated. We conclude that LLMs are able to generate convincing news articles that agree with dangerous disinformation narratives.

Motivation & Objective

• To assess the extent to which current instruction-tuned LLMs can generate convincing disinformation news articles in English.
• To evaluate the effectiveness of safety filters in preventing the generation of harmful disinformation.
• To measure the detectability of LLM-generated disinformation using existing detection models.
• To explore the feasibility of automating the evaluation of disinformation capabilities using LLMs.
• To provide a benchmark for understanding the current state of LLMs as tools for large-scale disinformation.

Proposed method

• The study prompted 10 LLMs to generate news articles based on 20 predefined disinformation narratives, including health-related hoaxes and conspiracy theories.
• Generated texts were manually evaluated by human annotators on agreement with narratives, use of novel arguments, and stylistic coherence.
• Safety filters were assessed by measuring the frequency of disclaimers, counterarguments, or refusal to generate content.
• Detection models were tested on their ability to identify LLM-generated disinformation with high precision.
• A subset of generated texts was evaluated using GPT-4 to assess the feasibility of automated evaluation pipelines.
• The evaluation was conducted using a standardized prompt framework and a controlled annotation process to ensure consistency.

Figure 1: Summary of how many generated texts we consider dangerous or safe . Dangerous texts are proper disinformation articles that could be misused. Safe texts contain disclaimers, provide counterarguments, argue against the user, etc. Note that GPT-4 annotations are generally biased towards safe

Experimental results

Research questions

• RQ1 To what extent can current LLMs generate news articles that align with dangerous disinformation narratives?
• RQ2 How effective are safety filters in preventing LLMs from generating harmful disinformation content?
• RQ3 How detectable are LLM-generated disinformation articles using existing detection models?
• RQ4 Can LLMs be used to automate the evaluation of disinformation capabilities in other LLMs?
• RQ5 How do different LLMs vary in their disinformation generation capabilities and safety compliance?

Key findings

• LLMs, including open-source models, can generate highly convincing disinformation news articles that align with dangerous narratives, such as health hoaxes and conspiracy theories.
• Safety filters in most LLMs failed to prevent the generation of harmful content, with only a few models showing notable resistance to disinformation prompts.
• A significant proportion of generated texts (as shown in Figure 1) were classified as 'dangerous'—meaning they fully supported disinformation without disclaimers or counterarguments.
• Detection models were able to identify LLM-generated disinformation with high precision, indicating that automated detection remains a viable defense mechanism.
• GPT-4 was able to partially automate the evaluation process, suggesting that LLM-based evaluation pipelines could scale future assessments with minimal human labor.
• The study highlights that prompt engineering by malicious actors could further improve disinformation quality and bypass safety mechanisms, indicating that current evaluations may underestimate real-world risks.

Figure 2: The average score for each question and LLM using (a) human and (b) GPT-4 annotations.