[Paper Review] GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysis

Yuqiang Sun, Daoyuan Wu+6 more|arXiv (Cornell University)|Aug 7, 2023

Software Engineering ResearchComputer Science36 references14 citations

TL;DR

GPTScan combines GPT-based code understanding with static analysis to detect logic vulnerabilities in Solidity smart contracts, achieving high precision and recall across multiple datasets while reducing false positives through static confirmation.

ABSTRACT

Smart contracts are prone to various vulnerabilities, leading to substantial financial losses over time. Current analysis tools mainly target vulnerabilities with fixed control or data-flow patterns, such as re-entrancy and integer overflow. However, a recent study on Web3 security bugs revealed that about 80% of these bugs cannot be audited by existing tools due to the lack of domain-specific property description and checking. Given recent advances in Large Language Models (LLMs), it is worth exploring how Generative Pre-training Transformer (GPT) could aid in detecting logicc vulnerabilities. In this paper, we propose GPTScan, the first tool combining GPT with static analysis for smart contract logic vulnerability detection. Instead of relying solely on GPT to identify vulnerabilities, which can lead to high false positives and is limited by GPT's pre-trained knowledge, we utilize GPT as a versatile code understanding tool. By breaking down each logic vulnerability type into scenarios and properties, GPTScan matches candidate vulnerabilities with GPT. To enhance accuracy, GPTScan further instructs GPT to intelligently recognize key variables and statements, which are then validated by static confirmation. Evaluation on diverse datasets with around 400 contract projects and 3K Solidity files shows that GPTScan achieves high precision (over 90%) for token contracts and acceptable precision (57.14%) for large projects like Web3Bugs. It effectively detects ground-truth logic vulnerabilities with a recall of over 70%, including 9 new vulnerabilities missed by human auditors. GPTScan is fast and cost-effective, taking an average of 14.39 seconds and 0.01 USD to scan per thousand lines of Solidity code. Moreover, static confirmation helps GPTScan reduce two-thirds of false positives.

Motivation & Objective

• Address the gap where most vulnerabilities are logic-based and not captured by pattern-driven tools.
• Leverage GPT as a code understanding tool to identify code-level scenarios and properties of vulnerabilities.
• Combine GPT-based matching with static confirmation to validate candidate vulnerabilities.
• Evaluate GPTScan on diverse real-world datasets to assess precision, recall, and cost-efficiency.

Proposed method

• Decompose logic vulnerability types into code-level scenarios and properties for GPT matching.
• Apply multi-dimensional filtering to narrow down candidate functions before GPT analysis.
• Use GPT-3.5-turbo with a scenario-then-property matching prompt to identify likely vulnerable functions.
• Prompt GPT to extract key variables and statements for static confirmation.
• Validate GPT-identified candidates with four static analysis modules: Data Flow Tracing, Value Comparison Check, Order Check, and Function Call Argument Check.

Experimental results

Research questions

• RQ1 Can GPT, used as a general code understanding tool, effectively identify logic vulnerabilities when combined with static analysis?
• RQ2 How can multi-dimensional filtering reduce GPT costs while maintaining detection accuracy across large Solidity projects?
• RQ3 What is the impact of static confirmation on reducing GPT false positives and improving precision?
• RQ4 How does GPTScan perform on token contracts versus large multi-file projects in terms of precision and recall?

Key findings

• GPTScan achieves high precision (over 90%) on token contracts (Top200 and DefiHacks datasets).
• GPTScan achieves acceptable precision (57.14%) on large projects like Web3Bugs.
• Ground-truth logic vulnerabilities are detected with recall over 70%, including 9 new vulnerabilities not reported by human auditors.
• Average scan time is 14.39 seconds and cost is 0.01 USD per thousand lines of Solidity code; larger projects incur higher cost and slower scans (~0.018 USD and ~20 seconds).
• Static confirmation reduces 65.84% of original false positives in the Web3Bugs dataset.