[Paper Review] Common Vulnerability Scoring System Prediction based on Open Source Intelligence Information Sources

Philipp Kuehn, David N. Relke+1 more|arXiv (Cornell University)|Oct 5, 2022

Web Application Security Vulnerabilities4 citations

TL;DR

This paper proposes a deep learning approach to predict Common Vulnerability Scoring System (CVSS) vectors using open-source intelligence (OSINT) texts beyond traditional vulnerability databases like NVD. By web scraping and analyzing referenced web pages—such as blogs, news articles, and patch notes—the study trains DistilBERT-based models on combined NVD descriptions and OSINT texts, achieving state-of-the-art performance in CVSS component prediction despite no significant improvement from OSINT data alone.

ABSTRACT

The number of newly published vulnerabilities is constantly increasing. Until now, the information available when a new vulnerability is published is manually assessed by experts using a Common Vulnerability Scoring System (CVSS) vector and score. This assessment is time consuming and requires expertise. Various works already try to predict CVSS vectors or scores using machine learning based on the textual descriptions of the vulnerability to enable faster assessment. However, for this purpose, previous works only use the texts available in databases such as National Vulnerability Database. With this work, the publicly available web pages referenced in the National Vulnerability Database are analyzed and made available as sources of texts through web scraping. A Deep Learning based method for predicting the CVSS vector is implemented and evaluated. The present work provides a classification of the National Vulnerability Database's reference texts based on the suitability and crawlability of their texts. While we identified the overall influence of the additional texts is negligible, we outperformed the state-of-the-art with our Deep Learning prediction models.

Motivation & Objective

• To investigate whether publicly available web sources beyond vulnerability databases can improve CVSS vector prediction.
• To identify and categorize OSINT sources (e.g., blogs, news, GitHub) that report on vulnerabilities.
• To build and evaluate a deep learning model using both NVD descriptions and scraped OSINT texts for CVSS prediction.
• To assess the feasibility and quality of using OSINT as training data for automated vulnerability severity scoring.
• To compare the performance of the proposed model against existing state-of-the-art methods using standardized metrics.

Proposed method

• Web scraping was applied to extract textual content from URLs referenced in NVD entries, focusing on domains like blogs, news sites, and GitHub.
• Texts from scraped sources were filtered, cleaned, and combined with official NVD vulnerability descriptions for model training.
• A fine-tuned DistilBERT model was used as the core deep learning architecture for multi-label classification of CVSS vector components.
• The model was trained and evaluated using metrics including Mean Squared Error (MSE) and Mean Absolute Error (MAE) on multiple test sets.
• The study evaluated the impact of OSINT data by comparing models trained on NVD-only data versus those including OSINT texts.
• A comparative evaluation was conducted against prior work, though limitations in model availability and metric consistency hindered full reproducibility.

Experimental results

Research questions

• RQ1 Where can relevant textual information on vulnerabilities be found outside traditional vulnerability databases (RQ1)?
• RQ2 To which degree are public data sources beyond vulnerability databases suitable for predicting the CVSS vector (RQ2)?
• RQ3 Can OSINT-derived texts improve the accuracy of CVSS vector prediction models compared to NVD-only training data?
• RQ4 Which types of OSINT sources are most reliable and crawlable for collecting vulnerability-related textual data?
• RQ5 How do different deep learning models perform in predicting individual CVSS vector components and overall severity scores?

Key findings

• The study successfully identified and categorized 11,738 unique URLs from NVD entries, classifying them into 12 distinct categories based on content and usability.
• Despite extensive scraping and integration of OSINT texts, the additional data did not yield a statistically significant improvement in model performance.
• The DistilBERT-based model achieved state-of-the-art results, with a Mean Absolute Error (MAE) of 0.203 on the Desc2022 test set when using only NVD descriptions.
• The model trained on combined NVD and OSINT data achieved an MAE of 0.248, which was not better than the NVD-only baseline, indicating no clear benefit from OSINT in this setup.
• The results suggest that while OSINT sources are usable for training, their noisy and inconsistent nature may limit their impact on model accuracy.
• The study highlights the need for better text filtering and domain-specific pretraining to unlock the potential of OSINT in CVSS prediction.