[Paper Review] D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysis

Yunhui Zheng, Saurabh Pujar+7 more|arXiv (Cornell University)|Feb 16, 2021

Software Engineering Research34 references10 citations

TL;DR

D2A proposes a differential analysis-based method to automatically label static analysis issues as true positives or false positives using bug-fixing commits in open-source projects, generating a large, high-quality dataset of over 1.3 million examples with detailed contextual information. The approach significantly improves label quality over static analysis alone and enables effective training of machine learning models to reduce false positives in vulnerability detection.

ABSTRACT

Static analysis tools are widely used for vulnerability detection as they understand programs with complex behavior and millions of lines of code. Despite their popularity, static analysis tools are known to generate an excess of false positives. The recent ability of Machine Learning models to understand programming languages opens new possibilities when applied to static analysis. However, existing datasets to train models for vulnerability identification suffer from multiple limitations such as limited bug context, limited size, and synthetic and unrealistic source code. We propose D2A, a differential analysis based approach to label issues reported by static analysis tools. The D2A dataset is built by analyzing version pairs from multiple open source projects. From each project, we select bug fixing commits and we run static analysis on the versions before and after such commits. If some issues detected in a before-commit version disappear in the corresponding after-commit version, they are very likely to be real bugs that got fixed by the commit. We use D2A to generate a large labeled dataset to train models for vulnerability identification. We show that the dataset can be used to build a classifier to identify possible false alarms among the issues reported by static analysis, hence helping developers prioritize and investigate potential true positives first.

Motivation & Objective

• To address the critical limitation of existing datasets for AI-based vulnerability detection, which suffer from synthetic content, limited context, or small scale.
• To develop an automated, scalable method for labeling static analysis issues as true positives or false positives using real-world software projects.
• To generate a high-quality, large-scale dataset with detailed contextual information (e.g., bug types, root causes, traces) for training robust machine learning models.
• To enable effective false positive reduction in static analysis by training models on the D2A-labeled dataset, improving developer productivity.

Proposed method

• Differential analysis is applied to version pairs (before and after bug-fixing commits) from six large open-source projects.
• Static analysis is run on both versions; issues present in the 'before' version but absent in the 'after' version are labeled as likely true positives.
• The method leverages commit history and inter-procedural static analysis to extract detailed bug context, including root cause locations and traces.
• A multi-project dataset is constructed by aggregating labeled examples across projects, with train/dev/test splits preserved per project.
• Machine learning models (e.g., CatBoost, LightGBM, Random Forest, Extra-Trees) are trained on the D2A dataset to classify static analysis issues as true positives or false positives.
• Ensemble voting across models is applied to improve generalization and performance, especially in combined evaluation across projects.

Experimental results

Research questions

• RQ1 Can differential analysis of version pairs reliably identify true positives among static analysis reports?
• RQ2 Can a large-scale, automated labeling approach produce higher-quality labels than static analysis alone?
• RQ3 Can the D2A dataset effectively train machine learning models to reduce false positives in static analysis?
• RQ4 How does model performance vary across different software projects, and can ensemble methods improve generalization?

Key findings

• The D2A dataset contains over 1.3 million labeled examples from six large open-source projects, with detailed contextual information including bug types, root causes, and traces.
• Manual validation confirms that D2A significantly improves label quality compared to static analysis alone, reducing ambiguity and false positives in labeling.
• The D2A-based false positive reduction model achieves a mean F1 score of 0.54 and a False Positive Reduction Rate (FPRR) of 83.7% on the combined dataset, outperforming individual models.
• Ensemble voting across multiple models (e.g., CatBoost, LightGBM, Random Forest, Extra-Trees) achieves the best performance, with an F1 score of 0.54 and FPRR of 83.7% on the combined dataset.
• On individual projects, the model achieves high FPRR (e.g., 97.3% on libtiff, 96.6% on libtiff with LightGBM), demonstrating strong performance on diverse codebases.
• The results show that the D2A dataset enables effective training of models that can prioritize true positives, reducing developer effort in triaging static analysis reports.