[Paper Review] Prompt Injection attack against LLM-integrated Applications

Yi Liu, Gelei Deng+10 more|arXiv (Cornell University)|Jun 8, 2023

Topic Modeling75 citations

TL;DR

The paper analyzes prompt injection risks in real-world LLM-integrated apps, introduces HouYi, a black-box attack framework, and validates it across 36 services with an 86.1% success rate, highlighting potential for prompt theft and misuse of LLM computation.

ABSTRACT

Large Language Models (LLMs), renowned for their superior proficiency in language comprehension and generation, stimulate a vibrant ecosystem of applications around them. However, their extensive assimilation into various services introduces significant security risks. This study deconstructs the complexities and implications of prompt injection attacks on actual LLM-integrated applications. Initially, we conduct an exploratory analysis on ten commercial applications, highlighting the constraints of current attack strategies in practice. Prompted by these limitations, we subsequently formulate HouYi, a novel black-box prompt injection attack technique, which draws inspiration from traditional web injection attacks. HouYi is compartmentalized into three crucial elements: a seamlessly-incorporated pre-constructed prompt, an injection prompt inducing context partition, and a malicious payload designed to fulfill the attack objectives. Leveraging HouYi, we unveil previously unknown and severe attack outcomes, such as unrestricted arbitrary LLM usage and uncomplicated application prompt theft. We deploy HouYi on 36 actual LLM-integrated applications and discern 31 applications susceptible to prompt injection. 10 vendors have validated our discoveries, including Notion, which has the potential to impact millions of users. Our investigation illuminates both the possible risks of prompt injection attacks and the possible tactics for mitigation.

Motivation & Objective

• Understand the practical constraints of existing prompt injection techniques on real-world LLM-integrated applications.
• Develop a black-box prompt injection methodology inspired by SQL injection and XSS attacks (HouYi).
• Demonstrate the attack's feasibility and quantify risks across real services, including potential financial impact.
• Propose insights for defense and encourage robust design of LLM-integrated applications.

Proposed method

• Survey and pilot study of ten real-world LLM-integrated applications to assess baseline effectiveness of existing prompt injection techniques.
• Develop HouYi, a three-component payload (Framework Component, Separator Component, Disruptor Component) that injects malicious prompts while mimicking legitimate prompts.
• Use a three-phase HouYi workflow: Context Inference, Payload Generation, and Feedback for iterative refinement.
• Evaluate HouYi on 36 LLM-integrated applications to measure success rate and analyze failure cases.
• Compare defenses from open-source projects and assess their limitations against HouYi-generated payloads.

Experimental results

Research questions

• RQ1 RQ1: What are the patterns and limitations of existing prompt injection attacks in practice on real-world LLM-integrated applications?
• RQ2 RQ2: How exploitable are current systems when subjected to black-box prompt injection techniques?
• RQ3 RQ3: How effective is the proposed HouYi framework across diverse application categories?
• RQ4 RQ4: What defenses or design principles can mitigate prompt injection risks in LLM-integrated services?

Key findings

• Existing prompt injection methods showed limited success in a pilot on 10 commercial apps due to varied prompt usage and defensive formatting.
• HouYi achieves an 86.1% success rate across 36 tested LLM-integrated applications, including potential prompt theft and unauthorized use of LLM compute.
• The attacks can induce the LLM to treat malicious payloads as questions and exploit context separation; however, defenses using formatting rules or multi-step processes can limit effectiveness.
• Vendors including Notion confirmed the findings, indicating potential impact on millions of users and possible financial losses in the millions of USD.
• Defenses from open-source projects mitigate some attacks but remain vulnerable to HouYi-generated payloads; the study highlights the need for stronger, more robust defenses.