[Paper Review] Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data

Nahema Marchal, Rachel Xu+4 more|arXiv (Cornell University)|Jun 19, 2024

Ethics and Social Impacts of AI13 citations

TL;DR

The paper builds a taxonomy of GenAI misuse tactics from qualitative analysis of ~200 real-world incidents (Jan 2023–Mar 2024), focusing on exploitation of capabilities and system compromise across modalities.

ABSTRACT

Generative, multimodal artificial intelligence (GenAI) offers transformative potential across industries, but its misuse poses significant risks. Prior research has shed light on the potential of advanced AI systems to be exploited for malicious purposes. However, we still lack a concrete understanding of how GenAI models are specifically exploited or abused in practice, including the tactics employed to inflict harm. In this paper, we present a taxonomy of GenAI misuse tactics, informed by existing academic literature and a qualitative analysis of approximately 200 observed incidents of misuse reported between January 2023 and March 2024. Through this analysis, we illuminate key and novel patterns in misuse during this time period, including potential motivations, strategies, and how attackers leverage and abuse system capabilities across modalities (e.g. image, text, audio, video) in the wild.

Motivation & Objective

• Develop a taxonomy of GenAI misuse tactics grounded in academic literature and real-world observations.
• Differentiate misuse tactics into exploitation of GenAI capabilities vs. compromise of GenAI systems.
• Characterize prevalence, motivations, and multimodal patterns to inform safety and governance.
• Identify how attackers leverage outputs across modalities to achieve goals like manipulation, fraud, or harassment.

Proposed method

• Literature review of malicious GenAI use and grey literature.
• Qualitative analysis of ~200 media reports of GenAI misuse published between Jan 2023 and Mar 2024.
• Dual independent coding of reports to identify relevant misuse tactics, with consensus on disagreements.
• Mapping cases to actor goals, tactics, tools, and targets; enriching with Appendix A/B data.
• Two data collection streams: proprietary social listening tool plus manual search; deduplication to 191 cases.

Experimental results

Research questions

• RQ1 What misuse tactics do actors employ with GenAI tools and how are they categorized?
• RQ2 How do misuse tactics distribute across modalities (text, image, audio, video) and actor goals?
• RQ3 What are the common strategies that combine tactics to achieve goals like manipulation or monetization?
• RQ4 To what extent are real-world incidents attacks on GenAI systems versus exploitation of capabilities?
• RQ5 What implications do these patterns have for governance, safety evaluations, and mitigations?

Key findings

• Approximately 9 out of 10 documented cases involve exploiting GenAI capabilities rather than attacking models directly.
• Impersonation-related tactics (Impersonation, Sockpuppeting, Appropriated Likeness, NCII) and Falsification dominate, often with political or monetary motives.
• Most misuse relies on readily accessible GenAI features requiring minimal technical expertise, rather than sophisticated system-targeted attacks.
• Emerging lower-level misuse includes political outreach and advocacy that challenges authenticity and disclosure norms.
• Documented cases of actual system compromise are few (two real-world instances) and often tied to research demonstrations rather than deployed attacks.
• Monetization and scams (content farms, impersonation-based fraud) and harassment (NCII) comprise major misuse categories, with reach and “digital resurrections” emerging as newer patterns.