[Paper Review] Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoning

Gelei Deng, Yi Liu+4 more|arXiv (Cornell University)|Feb 13, 2024

Imbalanced Data Classification Techniques4 citations

TL;DR

Pandora introduces a novel indirect jailbreak attack on GPTs via Retrieval-Augmented Generation (RAG) poisoning, where maliciously crafted external knowledge sources are injected to manipulate LLM responses. The method achieves 64.3% success on GPT-3.5 and 34.8% on GPT-4 across four prohibited content scenarios, outperforming direct prompt attacks.

ABSTRACT

Large Language Models~(LLMs) have gained immense popularity and are being increasingly applied in various domains. Consequently, ensuring the security of these models is of paramount importance. Jailbreak attacks, which manipulate LLMs to generate malicious content, are recognized as a significant vulnerability. While existing research has predominantly focused on direct jailbreak attacks on LLMs, there has been limited exploration of indirect methods. The integration of various plugins into LLMs, notably Retrieval Augmented Generation~(RAG), which enables LLMs to incorporate external knowledge bases into their response generation such as GPTs, introduces new avenues for indirect jailbreak attacks. To fill this gap, we investigate indirect jailbreak attacks on LLMs, particularly GPTs, introducing a novel attack vector named Retrieval Augmented Generation Poisoning. This method, Pandora, exploits the synergy between LLMs and RAG through prompt manipulation to generate unexpected responses. Pandora uses maliciously crafted content to influence the RAG process, effectively initiating jailbreak attacks. Our preliminary tests show that Pandora successfully conducts jailbreak attacks in four different scenarios, achieving higher success rates than direct attacks, with 64.3\% for GPT-3.5 and 34.8\% for GPT-4.

Motivation & Objective

• To investigate indirect jailbreak attacks on LLMs, particularly GPTs, by exploiting the integration of Retrieval-Augmented Generation (RAG).
• To develop a novel attack vector—RAG Poisoning—that manipulates external knowledge bases to induce malicious LLM behavior.
• To demonstrate the feasibility and high success rate of jailbreaking GPTs through poisoned RAG content, surpassing direct attack methods.
• To evaluate the effectiveness of this attack across multiple prohibited content scenarios, including adult, harmful, privacy-violating, and illegal content.

Proposed method

• The attack constructs malicious knowledge bases containing adversarial content designed to trigger jailbreak behavior when retrieved.
• It leverages prompt engineering to guide the LLM into retrieving and utilizing the poisoned RAG content during response generation.
• The method integrates with GPTs by creating custom GPT instances that incorporate the tainted knowledge base, enabling end-to-end jailbreak execution.
• It uses a two-stage process: first, poisoning the RAG knowledge source; second, crafting prompts that trigger retrieval of the malicious content.
• The attack is evaluated across four prohibited content categories: adult, harmful, privacy-violating, and illegal content.
• The framework is designed to be reusable and scalable, allowing any user to deploy a jailbroken GPT via the constructed poisoned GPT instance.

Experimental results

Research questions

• RQ1 Can RAG poisoning be used as a viable indirect jailbreak vector for GPTs, bypassing traditional safety filters?
• RQ2 How does the success rate of RAG poisoning compare to direct jailbreak attacks on GPT-3.5 and GPT-4?
• RQ3 Which prohibited content category is most susceptible to RAG-based jailbreaks, and why?
• RQ4 How does model architecture (e.g., GPT-3.5 vs. GPT-4) influence the effectiveness of RAG poisoning?
• RQ5 Can the attack be consistently reproduced across multiple trials, indicating reliability and robustness?

Key findings

• Pandora achieved a 64.3% success rate in jailbreaking GPT-3.5 across four prohibited content scenarios, significantly outperforming direct attacks.
• The success rate for GPT-4 was 34.8%, still substantially higher than the 1.0% success rate of direct attacks on GPT-4.
• The privacy category was the most vulnerable, with a 35.3% average success rate across all models and scenarios.
• The attack demonstrated high repeatability and consistency across multiple experimental rounds, indicating reliability.
• GPT-4-powered chatbots and GPTs were more resilient than GPT-3.5 counterparts, suggesting improved alignment in newer models.
• The results confirm that RAG poisoning is a potent and scalable attack vector that exploits the synergy between LLMs and external knowledge retrieval.