[Paper Review] LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations

Catherine Tony, Markus Mutas+2 more|arXiv (Cornell University)|Mar 16, 2023

Software Engineering Research4 citations

TL;DR

LLMSecEval introduces a dataset of 150 natural language prompts describing security-critical code scenarios prone to vulnerabilities from MITRE's Top 25 CWEs, paired with secure code examples. The dataset enables systematic evaluation of LLM-generated code for security flaws using automated analysis via CodeQL, demonstrating that LLMs can produce vulnerable code even when prompted with natural language descriptions.

ABSTRACT

Large Language Models (LLMs) like Codex are powerful tools for performing code completion and code generation tasks as they are trained on billions of lines of code from publicly available sources. Moreover, these models are capable of generating code snippets from Natural Language (NL) descriptions by learning languages and programming practices from public GitHub repositories. Although LLMs promise an effortless NL-driven deployment of software applications, the security of the code they generate has not been extensively investigated nor documented. In this work, we present LLMSecEval, a dataset containing 150 NL prompts that can be leveraged for assessing the security performance of such models. Such prompts are NL descriptions of code snippets prone to various security vulnerabilities listed in MITRE's Top 25 Common Weakness Enumeration (CWE) ranking. Each prompt in our dataset comes with a secure implementation example to facilitate comparative evaluations against code produced by LLMs. As a practical application, we show how LLMSecEval can be used for evaluating the security of snippets automatically generated from NL descriptions.

Motivation & Objective

• To address the lack of standardized, natural language prompts for evaluating the security of code generated by large language models (LLMs).
• To provide a benchmark dataset that maps natural language descriptions to known security vulnerabilities from MITRE's Top 25 CWEs.
• To enable comparative evaluation of LLMs' security output by including secure reference implementations for each prompt.
• To support research in prompt engineering for secure code generation and automated vulnerability detection in LLM-generated code.
• To demonstrate the feasibility of using CodeQL to detect CWE-related vulnerabilities in LLM-generated code.

Proposed method

• The dataset was constructed by using Codex to generate natural language descriptions from existing CWE-related code snippets, ensuring alignment with real-world vulnerabilities.
• Prompts were curated based on MITRE’s 2021 Top 25 CWE list, focusing on 18 of the 25 most critical weaknesses.
• Each prompt was manually reviewed and refined for language fluency, expressiveness, adequacy, and conciseness using both linguistic and content-related metrics.
• A secure reference implementation was provided for each prompt to enable direct comparison with LLM-generated code.
• An application was built using OpenAI’s GPT-3 and Codex APIs to generate code from the prompts, followed by automated vulnerability detection using CodeQL queries targeting 18 CWEs.
• The evaluation pipeline integrates prompt input, LLM code generation, and static analysis via CodeQL to identify security flaws in generated code.

Figure 1: NL prompts creation process

Experimental results

Research questions

• RQ1 Can natural language prompts describing security-relevant functionality reliably trigger LLMs to generate code with known vulnerabilities?
• RQ2 To what extent do LLMs like GPT-3 and Codex produce secure code when prompted with NL descriptions of CWE-prone scenarios?
• RQ3 How effective is CodeQL in detecting CWE-related vulnerabilities in LLM-generated code when used in conjunction with a curated prompt dataset?
• RQ4 Can a language-agnostic, prompt-based benchmark improve the reproducibility and standardization of security evaluations for code-generation LLMs?
• RQ5 How do language fluency and prompt clarity affect the security outcomes of LLM-generated code?

Key findings

• All 150 prompts in LLMSecEval received a naturalness score of 4 or higher, indicating high fluency in English, with most scoring 4 or above on expressiveness.
• 138 out of 150 prompts achieved a content adequacy score of 3 or higher, indicating most prompts contained sufficient and relevant information for secure code generation.
• 135 out of 150 prompts scored 3 or higher on conciseness, suggesting most prompts avoided unnecessary background details.
• The application built on top of LLMSecEval successfully used CodeQL to detect 18 of the 25 Top 25 CWEs in code generated by GPT-3 and Codex, validating the dataset’s utility in automated security evaluation.
• The dataset enables direct comparison between LLM-generated code and secure reference implementations, revealing that LLMs often produce vulnerable code even when prompted with natural language descriptions.
• The study demonstrates that LLMs trained on public codebases may inherit and propagate security flaws present in their training data, especially when prompted with ambiguous or incomplete descriptions.

Figure 2: An example of NL prompt generated from a Python code snippet covering CWE-20 scenario in the Pearce et al. [ 12 ] dataset.