[Paper Review] Do you still need a manual smart contract audit?

Isaac David, Liyi Zhou+4 more|arXiv (Cornell University)|Jun 21, 2023

Internet Traffic Analysis and Secure E-votingComputer Science19 citations

TL;DR

The paper evaluates GPT-4-32k and Claude-v1.3-100k for automated security audits of 52 vulnerable DeFi smart contracts, finding a 40% vulnerability-hit rate with many false positives; mutation testing yields up to ~78.8% true positives, indicating LLMs can aid but not replace manual audits.

ABSTRACT

We investigate the feasibility of employing large language models (LLMs) for conducting the security audit of smart contracts, a traditionally time-consuming and costly process. Our research focuses on the optimization of prompt engineering for enhanced security analysis, and we evaluate the performance and accuracy of LLMs using a benchmark dataset comprising 52 Decentralized Finance (DeFi) smart contracts that have previously been compromised. Our findings reveal that, when applied to vulnerable contracts, both GPT-4 and Claude models correctly identify the vulnerability type in 40% of the cases. However, these models also demonstrate a high false positive rate, necessitating continued involvement from manual auditors. The LLMs tested outperform a random model by 20% in terms of F1-score. To ensure the integrity of our study, we conduct mutation testing on five newly developed and ostensibly secure smart contracts, into which we manually insert two and 15 vulnerabilities each. This testing yielded a remarkable best-case 78.7% true positive rate for the GPT-4-32k model. We tested both, asking the models to perform a binary classification on whether a contract is vulnerable, and a non-binary prompt. We also examined the influence of model temperature variations and context length on the LLM's performance. Despite the potential for many further enhancements, this work lays the groundwork for a more efficient and economical approach to smart contract security audits.

Motivation & Objective

• Assess feasibility of using large language models (LLMs) for security auditing of smart contracts.
• Identify vulnerability types detectable by LLMs and measure accuracy against known attacks.
• Analyze the impact of context length, temperature, and prompting on LLM performance.
• Evaluate robustness via mutation testing with newly created secure contracts containing injected vulnerabilities.

Proposed method

• Use GPT-4-32k and Claude-v1.3-100k APIs to perform single-shot binary classifications of 38 vulnerability types across 52 vulnerable DeFi contracts.
• Provide the smart contract source code as context and prompt the model to answer YES/NO for each vulnerability type.
• Aggregate results to compute true positives, false positives, true negatives, and false negatives.
• Conduct mutation testing by inserting 2 or 15 vulnerabilities into five newly created secure contracts and re-evaluating with binary and non-binary prompts.
• Examine the effects of context length (token limits) and temperature on model performance.
• Provide two chain-of-thought reasoning case studies demonstrating prompts and outcomes.

Experimental results

Research questions

• RQ1 Can GPT-4-32k and Claude-v1.3-100k reliably identify vulnerability types in DeFi smart contracts?
• RQ2 What is the hit rate of LLMs on known vulnerable contracts, and how many false positives do they produce?
• RQ3 How do context length and temperature affect LLM performance in smart contract auditing?
• RQ4 Does mutation testing reveal robustness of LLM-based audits against unseen vulnerabilities?

Key findings

• LLMs identified 40% of vulnerability types across 52 DeFi attacks.
• There were 1318 false positives overall, indicating substantial manual verification is still required.
• When combined, GPT-4-32k and Claude-v1.3-100k achieved a 58/146 vulnerability-type hit rate (40%), with an average F1 score 0.077 for GPT-4-32k and 0.076 for Claude-100k.
• Mutation testing on five synthetic contracts yielded up to 78.8% true positives for GPT-4-32k under certain conditions.
• Non-binary prompts generally produced higher true-positive rates than binary prompts, suggesting richer responses can aid vulnerability discovery.
• Longer context lengths tended to reduce performance, while Claude showed relatively better true positives in longer contexts.