[Paper Review] SPI: Automated Identification of Security Patches via Commits

Yaqin Zhou, Jing Kai Siow+3 more|arXiv (Cornell University)|May 30, 2021

Software Engineering Research53 references18 citations

TL;DR

This paper proposes SPI, a deep learning system that automatically identifies security patches in open-source software by analyzing commit messages and code revisions. Using a dual neural network architecture—on commit messages and code changes—SPI achieves an 87.93% F1-score on a manually verified dataset of 38,291 commits, demonstrating high accuracy and generalization in industrial-scale settings.

ABSTRACT

Security patches in open-source software, providing security fixes to identified vulnerabilities, are crucial in protecting against cyberattacks. Despite the National Vulnerability Database (NVD) publishes identified vulnerabilities, a vast majority of vulnerabilities and their corresponding security patches remain beyond public exposure, e.g., in the open-source libraries that are heavily relied on by developers. An extensive security patches dataset could help end-users such as security companies, e.g., building a security knowledge base, or researchers, e.g., aiding in vulnerability research. To curate security patches including undisclosed patches at a large scale and low cost, we propose a deep neural-network-based approach built upon commits of open-source repositories. We build security patch datasets that include 38,291 security-related commits and 1,045 CVE patches from four C libraries. We manually verify each commit, among the 38,291 security-related commits, to determine if they are security-related. We devise a deep learning-based security patch identification system that consists of two neural networks: one commit-message neural network that utilizes pretrained word representations learned from our commits dataset; and one code-revision neural network that takes code before and after revision and learns the distinction on the statement level. Our evaluation results show that our system outperforms SVM and K-fold stacking algorithm, achieving as high as 87.93% F1-score and precision of 86.24%. We deployed our pipeline and learned model in an industrial production environment to evaluate the generalization ability of our approach. The industrial dataset consists of 298,917 commits from 410 new libraries that range from a wide functionality. Our experiment results and observation proved that our approach identifies security patches effectively among open-sourced projects.

Motivation & Objective

• To address the challenge of identifying undisclosed security patches in open-source software, which often remain hidden despite being actively fixed.
• To build a large-scale, manually verified dataset of security-related commits to support training and evaluation of automated patch identification systems.
• To develop a deep learning-based system that leverages both commit messages and code revisions to improve detection accuracy beyond traditional machine learning methods.
• To evaluate the system’s generalization capability on real-world industrial-scale repositories beyond the initial training data.

Proposed method

• Construct a security patch dataset of 38,291 commits and 1,045 CVE-mapped patches from four C-language open-source libraries, manually verified for security relevance.
• Design a dual neural network architecture: one for commit messages using pretrained word embeddings from the commit dataset, and one for code revisions that compares pre- and post-revision code at the statement level.
• Train the commit-message network using contextual embeddings to capture semantic intent of security-related commit descriptions.
• Train the code-revision network to detect subtle changes in code that indicate security fixes, even when changes are minimal or syntactically minor.
• Combine predictions from both networks using a weighted fusion strategy to improve overall classification performance.
• Deploy the trained model in an industrial environment on 298,917 commits from 410 new open-source libraries to evaluate real-world generalization.

Experimental results

Research questions

• RQ1 Can a deep learning model effectively identify security patches in open-source repositories using only commit-level information, including messages and code changes?
• RQ2 How does the performance of a dual-network model combining commit messages and code revisions compare to traditional machine learning baselines like SVM and K-fold stacking?
• RQ3 To what extent can the model generalize to new, unseen open-source projects beyond the training dataset?
• RQ4 How do minor or implicit code changes affect the model’s ability to detect security patches, especially when commit messages are vague or missing?
• RQ5 Can the model detect security patches that are not publicly disclosed or listed in vulnerability databases like NVD?

Key findings

• The proposed SPI system achieves an F1-score of 87.93% and precision of 86.24% on the combined dataset, significantly outperforming SVM and K-fold stacking baselines.
• The commit-message network performs better than the code-revision network, as commit messages contain more discriminative and interpretable signals for security intent.
• The code-revision network struggles with low-similarity changes (e.g., <10% token overlap), indicating limitations in detecting subtle or semantic changes in code.
• The model generalizes well to industrial-scale data, successfully identifying security patches in 298,917 commits from 410 new open-source libraries.
• False negatives are more likely due to implicit or poorly worded commit messages, but the code-revision network helps mitigate this by detecting changes even when messages are ambiguous.
• The study confirms that training data bias—especially missing implicit patches—can affect model performance, suggesting the need for more diverse and high-quality labeled data.