[Paper Review] Assessing Prompt Injection Risks in 200+ Custom GPTs

Jiahao Yu, Yuhang Wu+4 more|arXiv (Cornell University)|Nov 20, 2023

Adversarial Robustness in Machine Learning4 citations

TL;DR

This study investigates prompt injection vulnerabilities in over 200 user-created custom GPTs on the OpenAI platform, demonstrating that adversarial prompts can extract system prompts and uploaded files with high success rates. The research reveals that current defensive measures, including defensive prompts and code interpreter disabling, are insufficient against sophisticated attacks, highlighting an urgent need for stronger security frameworks in customizable LLMs.

ABSTRACT

In the rapidly evolving landscape of artificial intelligence, ChatGPT has been widely used in various applications. The new feature - customization of ChatGPT models by users to cater to specific needs has opened new frontiers in AI utility. However, this study reveals a significant security vulnerability inherent in these user-customized GPTs: prompt injection attacks. Through comprehensive testing of over 200 user-designed GPT models via adversarial prompts, we demonstrate that these systems are susceptible to prompt injections. Through prompt injection, an adversary can not only extract the customized system prompts but also access the uploaded files. This paper provides a first-hand analysis of the prompt injection, alongside the evaluation of the possible mitigation of such attacks. Our findings underscore the urgent need for robust security frameworks in the design and deployment of customizable GPT models. The intent of this paper is to raise awareness and prompt action in the AI community, ensuring that the benefits of GPT customization do not come at the cost of compromised security and privacy.

Motivation & Objective

• To investigate the security risks of prompt injection in user-customized GPT models deployed on the OpenAI platform.
• To assess the feasibility of extracting system prompts and uploaded files through adversarial prompts in real-world custom GPTs.
• To evaluate the effectiveness of existing defense mechanisms, such as defensive prompts and code interpreter disablement, against sophisticated red-teaming attacks.
• To raise awareness in the AI community about the risks of data leakage and intellectual property theft via prompt injection in customizable LLMs.
• To promote the development of robust, long-term security solutions for customizable AI models beyond current heuristic defenses.

Proposed method

• The researchers designed and deployed a series of adversarial prompts to test over 200 custom GPTs available on the OpenAI GPT Store.
• They used a two-stage red-teaming approach: first scanning for file names and code interpreter presence, then launching targeted prompt injection attacks.
• A red-teaming evaluation was conducted with four expert attackers using varying prompt strategies to test system prompt extraction and file leakage.
• The study evaluated the impact of disabling the code interpreter on attack success rates, comparing results across GPTs with and without this feature.
• The researchers implemented responsible disclosure by notifying OpenAI prior to publication and deleting all extracted data post-experiment.
• The full dataset, attack prompts, and red-teaming results were open-sourced to enable reproducibility and further research.

Figure 1: Proposed prompt injection method to extract system prompts and files from custom GPTs.

Experimental results

Research questions

• RQ1 To what extent can adversarial prompts extract the system prompt from user-customized GPTs?
• RQ2 Can attackers reliably retrieve uploaded files, including their names and sizes, through prompt injection?
• RQ3 How effective are defensive prompts in mitigating system prompt extraction and file leakage in custom GPTs?
• RQ4 Does disabling the code interpreter significantly reduce the success rate of prompt injection attacks?
• RQ5 Can sophisticated adversarial prompts bypass state-of-the-art defense mechanisms in real-world attack scenarios?

Key findings

• Over 200 custom GPTs were tested, and the vast majority were vulnerable to system prompt extraction using adversarial prompts.
• The presence of a code interpreter significantly increased the success rate and reduced the number of attempts required for prompt injection attacks.
• Even when defensive prompts were used, experienced attackers could bypass them, indicating that such defenses are insufficient against determined adversaries.
• Disabling the code interpreter improved security, but did not eliminate the risk, as three out of four experts still succeeded in extracting system prompts after 10 attempts.
• The study confirmed that uploaded files, including their names and sizes, could be systematically discovered and exfiltrated via prompt injection.
• The researchers successfully extracted system prompts and file metadata from the DALL·E GPT, demonstrating the exploitability of these vulnerabilities in real-world models.

Figure 2: Privacy issues with OpenAI interfaces. In the left figure, we could exploit the information of filenames. In the right figure, we could know how the user designed the plugin prototype for the custom GPT.