[Paper Review] Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks

Daniel Kang, Xuechen Li+4 more|arXiv (Cornell University)|Feb 11, 2023

Topic Modeling27 citations

TL;DR

The paper demonstrates that instruction-following LLMs can be weaponized using standard security-attacks (obfuscation, code injection/payload splitting, virtualization) to bypass content filters and generate targeted malicious content, at low cost and with high convincingness.

ABSTRACT

Recent advances in instruction-following large language models (LLMs) have led to dramatic improvements in a range of NLP tasks. Unfortunately, we find that the same improved capabilities amplify the dual-use risks for malicious purposes of these models. Dual-use is difficult to prevent as instruction-following capabilities now enable standard attacks from computer security. The capabilities of these instruction-following LLMs provide strong economic incentives for dual-use by malicious actors. In particular, we show that instruction-following LLMs can produce targeted malicious content, including hate speech and scams, bypassing in-the-wild defenses implemented by LLM API vendors. Our analysis shows that this content can be generated economically and at cost likely lower than with human effort alone. Together, our findings suggest that LLMs will increasingly attract more sophisticated adversaries and attacks, and addressing these attacks may require new approaches to mitigations.

Motivation & Objective

• Motivate and quantify dual-use risks of instruction-following LLMs.
• Show that LLMs behave like programmable systems capable of returning “gadgets” for security attacks.
• Demonstrate attacks that bypass in-the-wild defenses and evaluate economic viability of malicious generations.
• Argue for new defenses drawing from traditional computer security.

Proposed method

• Modeling LLMs as programmable systems with opcodes (memory, assignment, branching).
• Designing obfuscation, code injection/payload splitting, and virtualization attacks adapted from classic security to prompts.
• Evaluating attacks against OpenAI defenses (input/output filters, useless generations) across multiple malicious-use scenarios.
• Quantifying bypass success rates and comparing generation convincingness across models.
• Performing an economic analysis comparing model-generated vs. human-generated content costs.
• Analyzing personalization and convincingness of malicious generations.

Experimental results

Research questions

• RQ1 Can instruction-following LLMs bypass in-the-wild defenses (input/output filters, useless generations) with simple attacks?
• RQ2 Are model-generated malicious generations convincing and context-dependent, including personalized scams?
• RQ3 What is the economic feasibility of conducting malicious text generation with LLMs compared to human effort?
• RQ4 How do different instruction-following LLMs compare in terms of convincingness, consistency, personalization, and fluency of malicious content?
• RQ5 What defenses or analogies from traditional computer security might mitigate these dual-use risks?

Key findings

• Obfuscation and virtualization attacks achieved 100% bypass of OpenAI defenses across tested scenarios.
• Indirection achieved 92% bypass rate overall when initial defenses were not triggered.
• ChatGPT and text-davinci-003 produced the most convincing and consistent malicious content among models tested.
• Malicious content could be personalized to demographic profiles with high levels of personalization, convincingness, and fluency, especially for higher-capability models.
• Economic analysis indicates model-generated malicious content can cost as little as $0.0064 to $0.016 per generation, potentially cheaper than human-generated text.
• Attacks are easily scalable via templating and can be combined for higher effectiveness.