[Paper Review] DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructions

Fangzhou Wu, Xiaogeng Liu+1 more|arXiv (Cornell University)|Dec 7, 2023

Software Engineering Research6 citations

TL;DR

DeceptPrompt introduces a novel adversarial attack framework that generates deceptive natural language instructions to induce Code LLMs into producing functionally correct but vulnerable code. Using a genetic evolution-based optimization with a split control loss, it achieves an average 50% increase in attack success rate (ASR) by injecting benign-looking prefixes/suffixes that preserve semantic meaning while triggering specific vulnerabilities across multiple LLMs.

ABSTRACT

With the advancement of Large Language Models (LLMs), significant progress has been made in code generation, enabling LLMs to transform natural language into programming code. These Code LLMs have been widely accepted by massive users and organizations. However, a dangerous nature is hidden in the code, which is the existence of fatal vulnerabilities. While some LLM providers have attempted to address these issues by aligning with human guidance, these efforts fall short of making Code LLMs practical and robust. Without a deep understanding of the performance of the LLMs under the practical worst cases, it would be concerning to apply them to various real-world applications. In this paper, we answer the critical issue: Are existing Code LLMs immune to generating vulnerable code? If not, what is the possible maximum severity of this issue in practical deployment scenarios? In this paper, we introduce DeceptPrompt, a novel algorithm that can generate adversarial natural language instructions that drive the Code LLMs to generate functionality correct code with vulnerabilities. DeceptPrompt is achieved through a systematic evolution-based algorithm with a fine grain loss design. The unique advantage of DeceptPrompt enables us to find natural prefix/suffix with totally benign and non-directional semantic meaning, meanwhile, having great power in inducing the Code LLMs to generate vulnerable code. This feature can enable us to conduct the almost-worstcase red-teaming on these LLMs in a real scenario, where users are using natural language. Our extensive experiments and analyses on DeceptPrompt not only validate the effectiveness of our approach but also shed light on the huge weakness of LLMs in the code generation task. When applying the optimized prefix/suffix, the attack success rate (ASR) will improve by average 50% compared with no prefix/suffix applying.

Motivation & Objective

• To investigate whether Code LLMs are robust against adversarial natural language instructions that preserve semantic meaning but induce vulnerable code.
• To develop a practical red-teaming method that mimics real-world user inputs to test the worst-case security behavior of Code LLMs.
• To evaluate the susceptibility of popular Code LLMs (e.g., CodeLlama, WizardCoder) to generating vulnerable code under semantically benign yet maliciously crafted prompts.
• To understand the impact of prompt structure—such as prefix/suffix placement, length, and group size—on the effectiveness of adversarial prompting.
• To demonstrate that even functionally correct code can be systematically manipulated into containing critical vulnerabilities through subtle natural language manipulation.

Proposed method

• DeceptPrompt employs a genetic evolution-based optimization process to generate adversarial natural language prefixes and suffixes that are semantically benign but induce specific vulnerabilities in generated code.
• It introduces a novel split control loss function that simultaneously enforces functional correctness of the generated code and injects a target vulnerability (e.g., CWE-79, CWE-200).
• The method uses a reference language model (GPT-2) to evaluate perplexity, ensuring that optimized prefixes remain semantically coherent and undetectable as adversarial inputs.
• It conducts ablation studies on key hyperparameters, including prefix/suffix length, location (prefix vs. suffix), and population group size, to analyze their impact on attack success.
• The framework is evaluated across multiple Code LLMs, including CodeLlama-7B, StarChat-15B, and WizardCoder variants, under diverse vulnerability targets and prompt configurations.
• It leverages a real-world deployment scenario by using pure natural language instructions, avoiding reliance on code snippets or soft prompts, to simulate practical user interactions.

Experimental results

Research questions

• RQ1 Can Code LLMs be induced to generate functionally correct but vulnerable code through adversarial natural language instructions that preserve semantic meaning?
• RQ2 How effective is DeceptPrompt in increasing the attack success rate (ASR) compared to baseline prompts without adversarial prefixes/suffixes?
• RQ3 What is the impact of prompt structure—such as length, location (prefix vs. suffix), and group size—on the performance of the adversarial attack?
• RQ4 Can the adversarial prefixes generated by DeceptPrompt remain semantically coherent and undetectable via standard metrics like perplexity?
• RQ5 To what extent are popular Code LLMs like CodeLlama and WizardCoder vulnerable to such attacks across different vulnerability types (e.g., CWE-79, CWE-200)?

Key findings

• DeceptPrompt increases the average attack success rate (ASR) by 50% compared to baseline prompts without adversarial prefixes or suffixes.
• When using a 10-token seed prefix template, DeceptPrompt achieves an ASR of 62.5%, which drops to 55% with a 6-token template and 35% with a 3-token template, indicating that longer templates improve performance.
• Placing the adversarial context as a suffix reduces ASR from 62.5% to 52.5% and increases the wrong functionality rate (WFR) from 0% to 5%, showing that proximity to the task description enhances attack effectiveness.
• Reducing the population group size from 100 to 64 decreases ASR to 45%, and further reducing it to 40 results in 42.5% ASR, indicating that a larger search space improves optimization success.
• The perplexity of optimized prefixes remains low, demonstrating that DeceptPrompt maintains semantic coherence and avoids detection by standard language quality metrics.
• All evaluated Code LLMs—CodeLlama-7B, StarChat-15B, WizardCoder-3B, and WizardCoder-15B—were successfully attacked, revealing widespread vulnerability to this form of adversarial prompting.