[Paper Review] Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition

Sander Schulhoff, Jeremy Pinto+8 more|arXiv (Cornell University)|Oct 24, 2023

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper introduces a large-scale global prompt hacking competition, HackAPrompt, to systematically expose systemic vulnerabilities in large language models (LLMs) through human-generated adversarial prompts. By collecting over 600,000 prompts targeting three state-of-the-art LLMs, the study empirically validates the feasibility of prompt hacking and establishes a comprehensive taxonomical ontology of attack patterns, revealing critical security risks in real-world LLM deployments.

ABSTRACT

Large Language Models (LLMs) are deployed in interactive contexts with direct user engagement, such as chatbots and writing assistants. These deployments are vulnerable to prompt injection and jailbreaking (collectively, prompt hacking), in which models are manipulated to ignore their original instructions and follow potentially malicious ones. Although widely acknowledged as a significant security threat, there is a dearth of large-scale resources and quantitative studies on prompt hacking. To address this lacuna, we launch a global prompt hacking competition, which allows for free-form human input attacks. We elicit 600K+ adversarial prompts against three state-of-the-art LLMs. We describe the dataset, which empirically verifies that current LLMs can indeed be manipulated via prompt hacking. We also present a comprehensive taxonomical ontology of the types of adversarial prompts.

Motivation & Objective

• To address the lack of large-scale, quantitative studies on prompt hacking in LLMs.
• To simulate real-world attack scenarios by enabling free-form human input attacks on production-like LLM deployments.
• To collect and analyze a massive dataset of adversarial prompts to understand the types and success rates of prompt injection attacks.
• To develop a systematic taxonomical ontology of prompt hacking techniques for improved model robustness and defense strategies.
• To provide a publicly available dataset on HuggingFace to support future research in LLM security and alignment.

Proposed method

• Organized a global, prize-incentivized prompt hacking competition to elicit adversarial prompts from over 2,800 participants.
• Collected more than 600,000 adversarial prompts targeting three state-of-the-art LLMs, including GPT-3.5-turbo and others.
• Designed a structured competition with multiple levels, including a complex 'Mad Sandwich Defense' level requiring token-limit manipulation.
• Automated prompt generation and optimization using techniques like character replacement, translation to Chinese, and token counting to bypass constraints.
• Developed a pseudocode-based automation pipeline (Algorithm 1) to dynamically fill user input to token limits while preserving attack intent.
• Constructed a comprehensive taxonomical ontology of adversarial prompt types based on observed attack patterns and strategies.

Figure 1: Uses of llm s often define the task via a prompt template (top left), which is combined with user input (bottom left). We create a competition to see if user input can overrule the original task instructions and elicit specific target output (right).

Experimental results

Research questions

• RQ1 What types of adversarial prompts are most effective in manipulating state-of-the-art LLMs?
• RQ2 How prevalent and scalable are prompt injection attacks in real-world, interactive LLM deployments?
• RQ3 What systematic patterns and strategies emerge from human-generated prompt hacking attempts?
• RQ4 To what extent can automation enhance the discovery and refinement of effective prompt attacks?
• RQ5 How can a large-scale, human-curated dataset of adversarial prompts improve the evaluation and defense of LLM security?

Key findings

• Over 600,000 adversarial prompts were successfully collected from 2,800+ participants, demonstrating the widespread feasibility of prompt hacking.
• The study empirically confirms that current LLMs are highly susceptible to prompt injection attacks, even when protected by complex prompt templates.
• A wide variety of attack strategies were identified, including 'playing along' with the prompt, code injection, bracket/quote manipulation, and decoy inputs.
• Automation significantly improved attack efficiency, especially in token-constrained environments like level 9, where dynamic input filling and character replacement increased success rates.
• Translating parts of the prompt to Chinese helped bypass certain LLM misinterpretations, suggesting model behavior is sensitive to input encoding and language context.
• The resulting dataset is one of the largest known collections of user inputs mapped exactly to specific model completions, enabling new training and evaluation opportunities for safer LLMs.

Figure 2: In the competition playground, competitors select the challenge they would like to try (top left) and the model to evaluate with (upper mid left). They see the challenge description (mid left) as well as the prompt template for the challenge (lower mid left). As they type their input in th