[Paper Review] Anatomy of an AI-powered malicious social botnet

Kai‐Cheng Yang, Filippo Menczer|arXiv (Cornell University)|Jul 30, 2023

Misinformation and Its ImpactsSocial Sciences72 references12 citations

TL;DR

This study documents a Twitter botnet dubbed fox8 that uses ChatGPT to generate harmful content, analyzes its coordinated behavior, and evaluates detection methods showing current LLM-content detectors struggle to distinguish these bots from humans.

ABSTRACT

Large language models (LLMs) exhibit impressive capabilities in generating realistic text across diverse subjects. Concerns have been raised that they could be utilized to produce fake content with a deceptive intention, although evidence thus far remains anecdotal. This paper presents a case study about a Twitter botnet that appears to employ ChatGPT to generate human-like content. Through heuristics, we identify 1,140 accounts and validate them via manual annotation. These accounts form a dense cluster of fake personas that exhibit similar behaviors, including posting machine-generated content and stolen images, and engage with each other through replies and retweets. ChatGPT-generated content promotes suspicious websites and spreads harmful comments. While the accounts in the AI botnet can be detected through their coordination patterns, current state-of-the-art LLM content classifiers fail to discriminate between them and human accounts in the wild. These findings highlight the threats posed by AI-enabled social bots.

Motivation & Objective

• Identify and characterize a real-world AI-powered social botnet on Twitter.
• Understand how LLMs are leveraged to create human-like content and fake personas.
• Assess the detectability of LLM-powered bots with existing content detectors and bot-detection tools.
• Provide data and insights to the research community for further investigation into AI-enabled botnets.

Proposed method

• Identify botnet accounts via self-revealing tweets mentioning AI language models.
• Annotate a sample of accounts to classify likely humans vs. bots and construct the fox8-23 benchmark dataset.
• Characterize profile, social-network structure, and content patterns (originals, replies, retweets) of fox8 bots vs. baseline humans.
• Evaluate detection approaches using Botometer, OpenAI AI Text Classifier, and GPTZero on bot vs. human content.
• Propose an account-level detector using average OpenAI detector scores on qualified tweets to classify LLM-powered bots.

Experimental results

Research questions

• RQ1 Do real-world Twitter accounts use LLM-powered content generation to form coordinated malicious botnets?
• RQ2 What are the behavioral and network patterns of such AI-powered bots compared to humans?
• RQ3 Can existing LLM-content detectors and bot detectors reliably distinguish these bots from humans in the wild?
• RQ4 Can a practical detection approach be built from LLM-generated content signals to identify LLM-powered bots?

Key findings

• A dense cluster of 1,140 fox8 bot accounts engage in following each other and in mutual replies/retweets.
• Bots post a mix of original tweets, replies, and retweets, with an average of 25.6% original, 36.1% replies, and 38.4% retweets/quotes (SDs provided).
• Three suspicious websites (cryptnomics.org, fox8.news, globaleconomics.news) are frequently shared, and accounts link to them as part of botnet signals.
• Self-revealing tweets indicate use of ChatGPT-like prompts, with 81.3% showing harmful/negative content instructions related to OpenAI guidelines; others show prohibitions or beyond-capability prompts.
• Bot-detection tools fail to identify fox8 bots: Botometer distributions are left-skewed with recall near zero at threshold 2.5.
• OpenAI’s AI text classifier at the tweet level can separate bot vs. human signals better than GPTZero for this dataset, enabling an account-level detector with an F1 score of 0.84 at threshold 52.7; random accounts show overlap, indicating false positives.