[Paper Review] From Threat Reports to Continuous Threat Intelligence: A Comparison of Attack Technique Extraction Methods from Textual Artifacts

Md Rayhanur Rahman, Laurie Williams|arXiv (Cornell University)|Oct 5, 2022

Cybercrime and Law Enforcement Studies4 citations

TL;DR

This study evaluates and compares five text-based TTP (Tactics, Techniques, and Procedures) extraction methods from cybersecurity threat reports using TF-IDF, LSI, and other NLP techniques. It finds that TF-IDF and LSI achieve the highest F1 scores (84% and 83%, respectively), and demonstrates that oversampling improves performance on imbalanced datasets, offering a benchmark for future TTP extraction research with open-source code and a dataset provided.

ABSTRACT

The cyberthreat landscape is continuously evolving. Hence, continuous monitoring and sharing of threat intelligence have become a priority for organizations. Threat reports, published by cybersecurity vendors, contain detailed descriptions of attack Tactics, Techniques, and Procedures (TTP) written in an unstructured text format. Extracting TTP from these reports aids cybersecurity practitioners and researchers learn and adapt to evolving attacks and in planning threat mitigation. Researchers have proposed TTP extraction methods in the literature, however, not all of these proposed methods are compared to one another or to a baseline. extit{The goal of this study is to aid cybersecurity researchers and practitioners choose attack technique extraction methods for monitoring and sharing threat intelligence by comparing the underlying methods from the TTP extraction studies in the literature.} In this work, we identify ten existing TTP extraction studies from the literature and implement five methods from the ten studies. We find two methods, based on Term Frequency-Inverse Document Frequency(TFIDF) and Latent Semantic Indexing (LSI), outperform the other three methods with a F1 score of 84\% and 83\%, respectively. We observe the performance of all methods in F1 score drops in the case of increasing the class labels exponentially. We also implement and evaluate an oversampling strategy to mitigate class imbalance issues. Furthermore, oversampling improves the classification performance of TTP extraction. We provide recommendations from our findings for future cybersecurity researchers, such as the construction of a benchmark dataset from a large corpus; and the selection of textual features of TTP. Our work, along with the dataset and implementation source code, can work as a baseline for cybersecurity researchers to test and compare the performance of future TTP extraction methods.

Motivation & Objective

• To compare the performance of existing TTP extraction methods from cybersecurity threat reports to guide practitioners and researchers in method selection.
• To evaluate the impact of class imbalance and increasing label counts on TTP classification performance.
• To assess the effectiveness of oversampling techniques in mitigating performance degradation due to imbalanced datasets.
• To establish a reproducible benchmark for future TTP extraction research by implementing and evaluating five representative methods.
• To recommend best practices for feature selection and model tuning in TTP extraction tasks.

Proposed method

• The study implements five TTP extraction methods from existing literature: M:LSI, M:TFIDF-NP, M:TFIDF, M:LIS-Co, and M:BM25, using the ATT&CK framework as the ground truth.
• Textual descriptions of attack procedures from threat reports are processed using TF-IDF, LSI, and other NLP-based feature extraction techniques to represent documents numerically.
• Six classifiers are applied with default hyperparameters to evaluate classification performance across different methods.
• A synthetic oversampling strategy (SMOTE) is applied to the numeric textual features to address class imbalance in the dataset.
• The study evaluates performance under increasing label counts (i.e., number of TTP classes) to simulate evolving threat landscapes.
• A benchmark dataset is constructed from threat reports, and all implementations and code are made publicly available on GitHub.

Experimental results

Research questions

• RQ1 RQ1: How do the TTP extraction methods perform in classifying textual descriptions of attack procedures to attack techniques across different classifiers?
• RQ2 RQ2: How does class imbalance affect the performance of TTP extraction methods, and does oversampling improve classification results?
• RQ3 RQ3: How does the performance of TTP extraction methods degrade as the number of classification labels increases?
• RQ4 RQ4: Which textual features are most effective for TTP classification, and what are the implications for future feature engineering?

Key findings

• TF-IDF and LSI-based methods achieved the highest F1 scores of 84% and 83%, respectively, outperforming other methods in TTP classification.
• Performance across all methods decreased when the number of classification labels increased, indicating scalability challenges with growing TTP taxonomies.
• Oversampling using SMOTE improved classification performance by reducing bias toward majority classes in imbalanced datasets.
• The study identifies TF-IDF as the most dominant feature for TTP classification, suggesting it should be prioritized in future feature engineering.
• The authors recommend constructing a standardized benchmark dataset and conducting hyperparameter tuning to further improve model performance.
• The dataset and source code for all five implemented methods are publicly available, enabling reproducibility and future method comparison.