[Paper Review] Evaluation of ChatGPT Model for Vulnerability Detection

Anton Cheshkov, P. Zadorozhny+1 more|arXiv (Cornell University)|Apr 12, 2023

Software Engineering ResearchComputer Science24 citations

TL;DR

The paper empirically evaluates ChatGPT and GPT-3 on Java code vulnerability detection and finds they do not outperform a dummy baseline in binary or multi-label classification on a real-world CWE-focused dataset.

ABSTRACT

In this technical report, we evaluated the performance of the ChatGPT and GPT-3 models for the task of vulnerability detection in code. Our evaluation was conducted on our real-world dataset, using binary and multi-label classification tasks on CWE vulnerabilities. We decided to evaluate the model because it has shown good performance on other code-based tasks, such as solving programming challenges and understanding code at a high level. However, we found that the ChatGPT model performed no better than a dummy classifier for both binary and multi-label classification tasks for code vulnerability detection.

Motivation & Objective

• Assess whether large language models (ChatGPT and GPT-3) can detect vulnerabilities in Java code.
• Create a real-world dataset of vulnerable and patched Java functions focusing on CWE types.
• Compare model performance against a baseline and analyze prompts and prompts variants.

Proposed method

• Construct a dataset of Java functions from open GitHub repositories with known vulnerabilities and patches.
• Prepare subsets for binary (vulnerable vs. patched) and multi-label (top CWE types) classification.
• Query GPT-3 and ChatGPT via OpenAI API with prompts designed to elicit Yes/No vulnerability assessments and CWE-type labels.
• Evaluate using accuracy, precision, recall, F1, and AUC for binary, plus per-CWE metrics for multi-label tasks.
• Compare against a baseline dummy classifier and analyze prompt variations and temperature settings (set to 0).

Figure 1: Function size distribution in the dataset.

Experimental results

Research questions

• RQ1 Can ChatGPT and GPT-3 reliably detect vulnerabilities in Java code?
• RQ2 Do these models outperform a baseline dummy classifier on binary vulnerability detection?
• RQ3 How do the models perform on multi-label CWE-type classification for the top frequent vulnerabilities?
• RQ4 What is the impact of prompt design on vulnerability detection performance?
• RQ5 Do chain-of-thought prompting or larger models improve results in this task?],
• RQ6 key_findings':['Binary classification results show limited performance; text-davinci-003 achieves precision 0.50, recall 0.99, F1 0.67, AUC 0.51; gpt-3.5-turbo achieves precision 0.51, recall 0.80, F1 0.62, AUC 0.51; baseline remains at 0.50 across metrics.
• RQ7 Overall multi-label classification on the top five CWE types yields around 38% accuracy for GPT-3 / ChatGPT variants, with macro and weighted averages typically below 0.3 for many CWE types.
• RQ8 The negative (non-vulnerable) class dominates the samples, and models tend to over-predict positives, inflating false positives.
• RQ9 When excluding negative examples, accuracy decreases, indicating the models struggle to correctly identify vulnerable functions without the context of non-vulnerable ones.
• RQ10 Prompts exploring multiple phrasings do not yield substantial performance gains, suggesting prompt variation did not meaningfully improve results.
• RQ11 An anomaly observed: ChatGPT predicted vulnerability types more accurately for fixed functions, suggesting dataset or training biases may influence labeling.