[Paper Review] Ignore Previous Prompt: Attack Techniques For Language Models

Fábio Perez, Ian Ribeiro|arXiv (Cornell University)|Nov 17, 2022

Adversarial Robustness in Machine Learning74 citations

TL;DR

This paper introduces PromptInject, a framework to study prompt injection attacks on LLMs, detailing two attack types—goal hijacking and prompt leaking—and provides empirical results showing significant susceptibility of GPT-3 family models to these attacks under various settings.

ABSTRACT

Transformer-based large language models (LLMs) provide a powerful foundation for natural language tasks in large-scale customer-facing applications. However, studies that explore their vulnerabilities emerging from malicious user interaction are scarce. By proposing PromptInject, a prosaic alignment framework for mask-based iterative adversarial prompt composition, we examine how GPT-3, the most widely deployed language model in production, can be easily misaligned by simple handcrafted inputs. In particular, we investigate two types of attacks -- goal hijacking and prompt leaking -- and demonstrate that even low-aptitude, but sufficiently ill-intentioned agents, can easily exploit GPT-3's stochastic nature, creating long-tail risks. The code for PromptInject is available at https://github.com/agencyenterprise/PromptInject.

Motivation & Objective

• Motivate and formalize the study of adversarial prompt attacks on LLMs in production-like settings.
• Propose PromptInject as a modular framework to compose and evaluate such attacks.
• Characterize feasible attack vectors (goal hijacking and prompt leaking) and identify factors influencing attack success.
• Provide empirical results across prompts and model settings to inform safer prompt-design practices.

Proposed method

• Define a modular prompt-composition framework (PromptInject) with base prompts, attack prompts, private values, and delimiters.
• Experimentally evaluate goal hijacking and prompt leaking using text-davinci-002 across 35 base prompts with multiple hyperparameters.
• Measure attack success with exact-match for goal hijacking and containment of the original prompt for prompt leaking, plus fuzzy-matching considerations.
• Analyze factors affecting attack effectiveness (e.g., delimiters, rogue strings, temperature, stop sequences, text after user_input) across model settings.
• Report a risk analysis (x-risk) and discuss potential mitigation avenues like content moderation or dual-parameter model designs.

Experimental results

Research questions

• RQ1 How susceptible are LLMs to adversarial prompt injections that hijack the goal of a prompt (goal hijacking) or leak the original prompt (prompt leaking)?
• RQ2 What prompt design or model settings increase or decrease the success rate of such attacks?
• RQ3 To what extent can production-grade prompts and settings withstand simple handcrafted adversarial inputs?
• RQ4 What practical mitigations could reduce these prompt-injection risks in deployed LLM applications.

Key findings

• Goal hijacking achieved about 58.6%±1.6% success under certain prompts and settings, illustrating high susceptibility of strong LLMs to malicious inputs.
• Prompt leaking achieved about 23.6%±2.7% success under optimized tweaks, indicating leakage is harder than hijacking but still feasible.
• Delimiters that clearly separate instructions significantly boost attack success; some delimiter lengths/repetitions maximize impact.
• Higher temperature slightly reduces attack success but increases model unpredictability; higher top-p/penalties have mixed or limited effects.
• Text-davinci-002 was notably more vulnerable to attacks than smaller/OpenAI models tested.
• Mitigation methods identified include stop sequences, restricting output length, post-processing moderation, and possibly architectural changes like dual-parameter safety controls.