[Paper Review] SkipAnalyzer: A Tool for Static Code Analysis with Large Language Models

Mohammad Mahdi Mohajer, Reem Aleithan+5 more|arXiv (Cornell University)|Oct 27, 2023

Software Engineering Research4 citations

TL;DR

SkipAnalyzer is an LLM-powered tool that automates static code analysis by detecting bugs, filtering false positives, and generating syntactically correct patches using ChatGPT. It achieves up to 76.95% accuracy in detecting Resource Leak bugs and 97.30% success in generating correct patches, significantly outperforming baseline tools like Infer in precision and repair accuracy.

ABSTRACT

We introduce SkipAnalyzer, a large language model (LLM)-powered tool for static code analysis. SkipAnalyzer has three components: 1) an LLM-based static bug detector that scans source code and reports specific types of bugs, 2) an LLM-based false-positive filter that can identify false-positive bugs in the results of static bug detectors (e.g., the result of step 1) to improve detection accuracy, and 3) an LLM-based patch generator that can generate patches for the detected bugs above. As a proof-of-concept, SkipAnalyzer is built on ChatGPT, which has exhibited outstanding performance in various software engineering tasks. To evaluate SkipAnalyzer, we focus on two types of typical and critical bugs that are targeted by static bug detection, i.e., Null Dereference and Resource Leak as subjects. We employ Infer to aid the gathering of these two bug types from 10 open-source projects. Consequently, our experiment dataset contains 222 instances of Null Dereference bugs and 46 instances of Resource Leak bugs. Our study demonstrates that SkipAnalyzer achieves remarkable performance in the mentioned static analysis tasks, including bug detection, false-positive warning removal, and bug repair. In static bug detection, SkipAnalyzer achieves accuracy values of up to 68.37% for detecting Null Dereference bugs and 76.95% for detecting Resource Leak bugs, improving the precision of the current leading bug detector, Infer, by 12.86% and 43.13%, respectively. For removing false-positive warnings, SkipAnalyzer can reach a precision of up to 93.88% for Null Dereference bugs and 63.33% for Resource Leak bugs. Additionally, SkipAnalyzer surpasses state-of-the-art false-positive warning removal tools. Furthermore, in bug repair, SkipAnalyzer can generate syntactically correct patches to fix its detected bugs with a success rate of up to 97.30%.

Motivation & Objective

• To develop an end-to-end, LLM-powered tool that automates static code analysis without human intervention.
• To address the high rate of false positives in static bug detectors like Infer, which require costly manual validation.
• To improve the precision and accuracy of static analysis by integrating LLM-based components for detection, filtering, and repair.
• To evaluate the effectiveness of LLMs in real-world software engineering tasks such as bug detection and patch generation.
• To demonstrate that LLMs can surpass traditional static analysis tools in key metrics like precision and repair success rate.

Proposed method

• SkipAnalyzer employs three LLM-based components: a static bug detector, a false-positive filter, and a patch generator, each using fine-tuned prompts.
• It leverages OpenAI’s ChatGPT-3.5 Turbo and ChatGPT-4 models with zero-shot, one-shot, and few-shot prompting strategies to enhance performance.
• The tool uses Infer to identify and label 222 Null Dereference and 46 Resource Leak bugs across 10 open-source Java projects, forming a labeled dataset.
• Prompt engineering is applied to maximize model performance, with context-aware and task-specific prompts tailored for each analysis component.
• The system evaluates model outputs for syntactic correctness and semantic accuracy, ensuring generated patches are valid and executable.
• Performance is measured using precision, accuracy, and success rate for detection, filtering, and repair tasks.

Experimental results

Research questions

• RQ1 Can an LLM-based system like SkipAnalyzer outperform traditional static analysis tools such as Infer in detecting Null Dereference and Resource Leak bugs?
• RQ2 To what extent can LLMs reduce false-positive warnings in static code analysis, especially when filtering outputs from existing tools?
• RQ3 Can LLMs generate syntactically correct and semantically valid patches for detected bugs with high success rates?
• RQ4 How do different prompting strategies (zero-shot, one-shot, few-shot) affect the performance of LLMs in static code analysis tasks?
• RQ5 Does the performance of SkipAnalyzer generalize across diverse open-source projects and bug types?

Key findings

• SkipAnalyzer achieves a detection accuracy of 68.37% for Null Dereference bugs and 76.95% for Resource Leak bugs, improving Infer’s precision by 12.86% and 43.13% respectively.
• The false-positive filter reaches a precision of 93.88% for Null Dereference and 63.33% for Resource Leak bugs, surpassing existing baselines.
• The patch generator achieves a 97.30% success rate in generating correct patches for Null Dereference bugs and 91.77% for Resource Leak bugs.
• 98.77% of all generated patches are syntactically correct, indicating high reliability in code generation.
• The false-positive filter improves the precision of SkipAnalyzer’s own detector by 16.31% for Null Dereference bugs, demonstrating internal consistency.
• SkipAnalyzer outperforms state-of-the-art tools in false-positive removal, with a 28.68% precision gain over Infer for Null Dereference bugs.