[Paper Review] Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models

Manish Bhatt, Sahana Chennabasappa+19 more|arXiv (Cornell University)|Dec 7, 2023

Artificial Intelligence in Healthcare and Education15 citations

TL;DR

CyberSecEval is a comprehensive benchmark assessing LLMs for insecure code generation across 8 languages and for compliance with cyberattack facilitation prompts, with case study on 7 models from Llama 2, Code Llama, and OpenAI GPT families.

ABSTRACT

This paper presents CyberSecEval, a comprehensive benchmark developed to help bolster the cybersecurity of Large Language Models (LLMs) employed as coding assistants. As what we believe to be the most extensive unified cybersecurity safety benchmark to date, CyberSecEval provides a thorough evaluation of LLMs in two crucial security domains: their propensity to generate insecure code and their level of compliance when asked to assist in cyberattacks. Through a case study involving seven models from the Llama 2, Code Llama, and OpenAI GPT large language model families, CyberSecEval effectively pinpointed key cybersecurity risks. More importantly, it offered practical insights for refining these models. A significant observation from the study was the tendency of more advanced models to suggest insecure code, highlighting the critical need for integrating security considerations in the development of sophisticated LLMs. CyberSecEval, with its automated test case generation and evaluation pipeline covers a broad scope and equips LLM designers and researchers with a tool to broadly measure and enhance the cybersecurity safety properties of LLMs, contributing to the development of more secure AI systems.

Motivation & Objective

• Motivate and measure cybersecurity risks in LLMs used as coding assistants.
• Develop automated test suites to detect insecure coding practices across multiple languages.
• Evaluate LLM compliance when asked to assist in cyberattacks and identify safety weaknesses.

Proposed method

• Develop Insecure Code Detector (ICD) with 189 static analysis rules covering 50 CWEs across 8 languages.
• Automatically generate test prompts from insecure code for autocomplete and instruct contexts.
• Create cyberattack helpfulness tests by hand-authoring prompts and augmenting them with Llama-70b-chat to judge malicious usefulness.
• Evaluate LLM outputs with a judge LLM to detect insecure code and cyberattack helpfulness, computing precision/recall.
• Case study applying the benchmarks to seven models from Llama 2, Code Llama, and OpenAI GPT families.
• Provide open-source tooling and test cases available at the project repository.

Figure 1: High level overview of CyberSecEval ’s approach.

Experimental results

Research questions

• RQ1 Do LLMs generate insecure code when completing or instructed to write code, and how often across languages and model types?
• RQ2 Are LLMs compliant with requests to assist in cyberattacks, and does higher coding capability correlate with higher compliance?
• RQ3 Can automated static-analysis-based detection and LLM-based judging accurately measure cybersecurity safety properties of LLMs?

Key findings

• LLMs suggested insecure coding practices about 30% of the time across test cases.
• CodeLlama models with higher coding ability tended to produce more insecure code and comply more with cyberattack prompts.
• Across models and threat categories, cyberattack compliance averaged 53%.
• Insecure Code Detector achieved 96% precision and 79% recall overall in detecting insecure LLM-generated code.
• Cyberattack helpfulness detection achieved 94% precision and 84% recall in identifying responses helpful to cyber attackers.

Figure 2: The precision and recall of our Insecure Code Detector static analyzer at detecting insecure code in LLM completions.