[Paper Review] Automated Mapping of CVE Vulnerability Records to MITRE CWE Weaknesses

Ashraf Haddad, Najwa Aaraj+2 more|arXiv (Cornell University)|Apr 13, 2023

Network Security and Intrusion Detection5 citations

TL;DR

This paper proposes a human-in-the-loop framework for automated mapping of CVE vulnerability records to MITRE CWE weaknesses using NLP, releasing a new dataset of 4,012 manually annotated records. It demonstrates that fine-tuned SBERT and rankT5 models significantly outperform BM25, BERT, and RoBERTa in semantic similarity ranking, highlighting the need for deep contextual understanding in vulnerability classification tasks.

ABSTRACT

In recent years, a proliferation of cyber-security threats and diversity has been on the rise culminating in an increase in their reporting and analysis. To counter that, many non-profit organizations have emerged in this domain, such as MITRE and OSWAP, which have been actively tracking vulnerabilities, and publishing defense recommendations in standardized formats. As producing data in such formats manually is very time-consuming, there have been some proposals to automate the process. Unfortunately, a major obstacle to adopting supervised machine learning for this problem has been the lack of publicly available specialized datasets. Here, we aim to bridge this gap. In particular, we focus on mapping CVE records into MITRE CWE Weaknesses, and we release to the research community a manually annotated dataset of 4,012 records for this task. With a human-in-the-loop framework in mind, we approach the problem as a ranking task and aim to incorporate reinforced learning to make use of the human feedback in future work. Our experimental results using fine-tuned deep learning models, namely Sentence-BERT and rankT5, show sizable performance gains over BM25, BERT, and RoBERTa, which demonstrates the need for an architecture capable of good semantic understanding for this task.

Motivation & Objective

• To address the lack of publicly available, specialized datasets for mapping CVE records to MITRE CWE weaknesses.
• To develop an automated, AI-assisted framework that supports human analysts in labeling vulnerabilities with standardized CWE types.
• To explore the use of reinforced learning with human feedback to iteratively improve model performance in vulnerability classification.
• To evaluate state-of-the-art deep learning models for their effectiveness in semantic similarity and ranking tasks on cyber-security text.
• To release a reproducible, open-source dataset and annotation pipeline to support future research in automated cyber-security reporting.

Proposed method

• Formulate the CVE-to-CWE mapping task as a ranking problem, where each CVE description is ranked against a set of CWE entries.
• Fine-tune sentence-transformer models (SBERT) and the rankT5 model for semantic similarity between CVE and CWE descriptions.
• Use BM25 as a baseline retrieval model based on lexical matching.
• Evaluate BERT and RoBERTa as baseline models for semantic similarity, despite their suboptimal design for ranking tasks.
• Train a T5-based sequence-to-sequence model for direct generation of CWE labels from CVE descriptions.
• Implement a human-in-the-loop feedback mechanism to support future integration with reinforcement learning from human feedback (RLHF).

Experimental results

Research questions

• RQ1 Can deep learning models achieve superior performance over traditional retrieval methods like BM25 in mapping unstructured CVE records to structured MITRE CWE weaknesses?
• RQ2 How do transformer-based models such as SBERT and rankT5 compare to general-purpose models like BERT and RoBERTa in this semantic ranking task?
• RQ3 To what extent can a sequence-to-sequence T5 model generate accurate CWE labels directly from CVE descriptions?
• RQ4 How effective is the proposed human-in-the-loop framework in improving model alignment with expert labeling decisions?
• RQ5 What is the impact of model architecture choice on performance when modeling semantic similarity between vulnerability and weakness descriptions?

Key findings

• Fine-tuned SBERT achieved the highest performance among all models, significantly outperforming BM25, BERT, and RoBERTa due to its specialized design for sentence similarity.
• The rankT5 model also demonstrated strong performance, showing that dedicated ranking architectures are effective for this task.
• The T5-based sequence-to-sequence model achieved a macro-average F1 score of 0.51 on generated CWE labels, with high accuracy (0.96) on frequent vulnerabilities like Cross-site Scripting.
• The model generated meaningful but incorrect labels for 7 previously unseen CWE types, indicating generative capability despite zero F1 on rare labels.
• The release of a 4,012-record manually annotated dataset fills a critical gap in cyber-security NLP research and enables future benchmarking and model training.
• The results confirm that semantic understanding is essential for this task, as lexical matching (BM25) is insufficient for accurate mapping.