[Paper Review] Multi-Level Fine-Tuning, Data Augmentation, and Few-Shot Learning for Specialized Cyber Threat Intelligence

Markus Bayer, Tobias Frey+1 more|arXiv (Cornell University)|Jul 22, 2022

Network Security and Intrusion Detection6 citations

TL;DR

This paper proposes a novel pipeline combining multi-level fine-tuning, GPT-3-based data augmentation, and few-shot learning to train high-accuracy cyber threat intelligence (CTI) classifiers from as few as 32 labeled instances. The approach achieves an F1 score within 5 points of a model trained on 1,800 instances, outperforming standard methods by over 21 points and state-of-the-art few-shot models by 18 points on a new expert-annotated dataset from the 2021 Microsoft Exchange Server breach.

ABSTRACT

Gathering cyber threat intelligence from open sources is becoming increasingly important for maintaining and achieving a high level of security as systems become larger and more complex. However, these open sources are often subject to information overload. It is therefore useful to apply machine learning models that condense the amount of information to what is necessary. Yet, previous studies and applications have shown that existing classifiers are not able to extract specific information about emerging cybersecurity events due to their low generalization ability. Therefore, we propose a system to overcome this problem by training a new classifier for each new incident. Since this requires a lot of labelled data using standard training methods, we combine three different low-data regime techniques - transfer learning, data augmentation, and few-shot learning - to train a high-quality classifier from very few labelled instances. We evaluated our approach using a novel dataset derived from the Microsoft Exchange Server data breach of 2021 which was labelled by three experts. Our findings reveal an increase in F1 score of more than 21 points compared to standard training methods and more than 18 points compared to a state-of-the-art method in few-shot learning. Furthermore, the classifier trained with this method and 32 instances is only less than 5 F1 score points worse than a classifier trained with 1800 instances.

Motivation & Objective

• To address the challenge of low generalization in existing cyber threat intelligence (CTI) classifiers when faced with novel, dynamic cyber threats.
• To reduce the labeling burden for training specialized CTI models by enabling high performance with minimal labeled data.
• To develop a robust, reusable framework for rapidly deploying accurate classifiers for emerging cyber threats using low-data regime techniques.
• To create a new, expert-annotated CTI dataset focused on a real-world cyber incident to support future research.

Proposed method

• The method employs multi-level transfer learning, starting from a large general-purpose language model and progressively fine-tuning it on increasingly specific cyber threat domains.
• Data augmentation is performed using GPT-3 with a prompting strategy, followed by human-in-the-loop filtering to preserve label fidelity and ensure semantic relevance.
• Few-shot learning is applied by training the model on only 32 labeled instances per threat class, leveraging meta-learning principles to generalize from minimal examples.
• The pipeline integrates transfer learning, data augmentation, and few-shot learning in a cascaded manner to progressively refine model performance.
• A human expert defines a boundary for filtering generated instances, discarding those too dissimilar to the original labeled data to maintain label consistency.
• The approach is evaluated on a new dataset derived from the 2021 Microsoft Exchange Server breach, annotated by three experts with high intercoder reliability.

Experimental results

Research questions

• RQ1 Can a combination of multi-level fine-tuning, data augmentation, and few-shot learning significantly improve F1 scores for specialized cyber threat intelligence classification with minimal labeled data?
• RQ2 How effective is GPT-3-based data augmentation with human-in-the-loop filtering in improving model robustness and performance when training data is extremely limited?
• RQ3 To what extent does multi-level fine-tuning outperform standard fine-tuning and state-of-the-art few-shot learning methods in low-data regimes for CTI?
• RQ4 How does the inclusion of data augmentation affect model variance and generalization, especially when combined with advanced fine-tuning and few-shot learning?
• RQ5 Can a new, expert-annotated dataset based on a real-world cyber incident serve as a reliable benchmark for future low-data CTI research?

Key findings

• The proposed method achieved an F1 score increase of over 21 points compared to standard training methods on the Microsoft Exchange Server breach dataset.
• The method outperformed a state-of-the-art few-shot learning model by more than 18 F1 points, demonstrating superior generalization from few examples.
• A classifier trained with only 32 labeled instances using the proposed pipeline achieved an F1 score within 5 points of a model trained on 1,800 instances.
• The inclusion of GPT-3-based data augmentation reduced model standard deviation, indicating improved robustness, though its impact on absolute F1 was modest when combined with multi-level fine-tuning.
• The expert-annotated dataset showed high intercoder reliability, validating its quality and utility for future research in specialized CTI.
• The method remains effective even when data augmentation is omitted, suggesting that multi-level fine-tuning and few-shot learning are the primary drivers of performance gains.