[Paper Review] On the Use of Fine-grained Vulnerable Code Statements for Software Vulnerability Assessment Models

Triet Huynh Minh Le, M. Ali Babar|arXiv (Cornell University)|Mar 16, 2022

Software Engineering Research4 citations

TL;DR

This paper proposes a machine learning approach to assess software vulnerability (SV) severity at the function level using fine-grained vulnerable code statements and their surrounding context. It demonstrates that vulnerable statements—though 5.8× smaller in size—yield 7.5–114.5% higher Matthews Correlation Coefficient (MCC) than non-vulnerable statements, with context integration boosting performance to 0.64 MCC and 0.75 F1-score, establishing strong baselines for pre-fixing SV assessment.

ABSTRACT

Many studies have developed Machine Learning (ML) approaches to detect Software Vulnerabilities (SVs) in functions and fine-grained code statements that cause such SVs. However, there is little work on leveraging such detection outputs for data-driven SV assessment to give information about exploitability, impact, and severity of SVs. The information is important to understand SVs and prioritize their fixing. Using large-scale data from 1,782 functions of 429 SVs in 200 real-world projects, we investigate ML models for automating function-level SV assessment tasks, i.e., predicting seven Common Vulnerability Scoring System (CVSS) metrics. We particularly study the value and use of vulnerable statements as inputs for developing the assessment models because SVs in functions are originated in these statements. We show that vulnerable statements are 5.8 times smaller in size, yet exhibit 7.5-114.5% stronger assessment performance (Matthews Correlation Coefficient (MCC)) than non-vulnerable statements. Incorporating context of vulnerable statements further increases the performance by up to 8.9% (0.64 MCC and 0.75 F1-Score). Overall, we provide the initial yet promising ML-based baselines for function-level SV assessment, paving the way for further research in this direction.

Motivation & Objective

• To address the delay in CVSS metric availability from NVD reports, which are typically published 146 days after vulnerability fixes.
• To investigate whether fine-grained vulnerable code statements and their context can improve function-level software vulnerability (SV) assessment.
• To develop data-driven models that predict seven CVSS metrics (e.g., Access Vector, Severity) directly from code, enabling early prioritization of fixes.
• To evaluate the relative effectiveness of vulnerable versus non-vulnerable code lines in predicting SV assessment metrics.
• To establish strong, reusable baselines for function-level SV assessment using real-world, diverse Java projects.

Proposed method

• The study uses 1,782 functions from 429 real-world vulnerabilities across 200 Java projects, extracted from public repositories.
• Vulnerable statements are identified using existing fine-grained SV detection methods, and their surrounding code context is preserved.
• Various feature extraction techniques (e.g., code embeddings, syntactic features) are applied to vulnerable statements and non-vulnerable lines.
• Multiple machine learning classifiers (e.g., Random Forest, XGBoost) are trained to predict seven CVSS metrics: Access Vector, Access Complexity, Authentication, Confidentiality, Integrity, Availability, and Severity.
• Performance is evaluated using Matthews Correlation Coefficient (MCC) and F1-Score, with statistical significance tested via Wilcoxon signed-rank tests.
• The model combines vulnerable statements with all other lines in the function to assess the impact of context on prediction performance.

Figure 2 . Methodology used to answer the research questions. Note: The vulnerable function is the one described in Fig. 1 .

Experimental results

Research questions

• RQ1 How effective are fine-grained vulnerable statements compared to non-vulnerable statements in predicting CVSS metrics for software vulnerabilities?
• RQ2 To what extent does incorporating contextual code around vulnerable statements improve the performance of function-level vulnerability assessment models?
• RQ3 Can machine learning models trained on code-level inputs achieve high performance in predicting CVSS metrics before vulnerability reports are published?
• RQ4 How does the size of vulnerable statements compare to non-vulnerable lines, and does this affect their predictive utility?
• RQ5 What is the generalizability of the proposed models across diverse real-world software projects?

Key findings

• Vulnerable statements are 5.8 times smaller in size (lines of code) than non-vulnerable statements, yet achieve 7.5–114.5% higher Matthews Correlation Coefficient (MCC) in predicting CVSS metrics.
• The integration of context around vulnerable statements increases model performance by up to 8.9%, achieving a peak MCC of 0.64 and F1-Score of 0.75.
• Vulnerable statements alone outperform non-vulnerable statements by 7.4–114.5% in MCC and 5.5–43.6% in F1-Score, indicating their high predictive relevance.
• The study's results are statistically significant with p-values < 0.01 and non-negligible effect sizes, confirming robustness across multiple metrics.
• The proposed models demonstrate strong performance even when trained on real-world Java projects across diverse domains and scales, suggesting practical applicability.
• The dataset and trained models are publicly released to support reuse and extension to other programming languages and applications.

Figure 3 . Class distributions of the seven CVSS metrics.