[Paper Review] A RAG-Based Question-Answering Solution for Cyber-Attack Investigation and Attribution

Sampath Rajapaksha, Ruby Rani+1 more|arXiv (Cornell University)|Aug 12, 2024

Topic Modeling4 citations

TL;DR

This paper presents the first Retrieval-Augmented Generation (RAG)-based question-answering system tailored for cyber-attack investigation and attribution, leveraging the AttackER knowledge base and a Mistral-based LLM to deliver accurate, source-referenced answers. It outperforms GPT-3.5 and GPT-4o by reducing hallucination and improving reliability through grounded responses.

ABSTRACT

In the constantly evolving field of cybersecurity, it is imperative for analysts to stay abreast of the latest attack trends and pertinent information that aids in the investigation and attribution of cyber-attacks. In this work, we introduce the first question-answering (QA) model and its application that provides information to the cybersecurity experts about cyber-attacks investigations and attribution. Our QA model is based on Retrieval Augmented Generation (RAG) techniques together with a Large Language Model (LLM) and provides answers to the users' queries based on either our knowledge base (KB) that contains curated information about cyber-attacks investigations and attribution or on outside resources provided by the users. We have tested and evaluated our QA model with various types of questions, including KB-based, metadata-based, specific documents from the KB, and external sources-based questions. We compared the answers for KB-based questions with those from OpenAI's GPT-3.5 and the latest GPT-4o LLMs. Our proposed QA model outperforms OpenAI's GPT models by providing the source of the answers and overcoming the hallucination limitations of the GPT models, which is critical for cyber-attack investigation and attribution. Additionally, our analysis showed that when the RAG QA model is given few-shot examples rather than zero-shot instructions, it generates better answers compared to cases where no examples are supplied in addition to the query.

Motivation & Objective

• To address the challenge of manual, time-intensive cyber-attack attribution in the face of unstructured, non-standardized threat intelligence reports.
• To reduce LLM hallucination and outdated knowledge in cyber threat analysis by grounding responses in a curated knowledge base.
• To develop a reliable, transparent QA system that supports cybersecurity analysts in investigating and attributing attacks with verifiable sources.
• To evaluate the performance of RAG-based QA on specialized cyber-attack attribution questions using multiple metrics.
• To explore the impact of few-shot prompting on answer quality in the context of cyber-attack investigation.

Proposed method

• The system uses a RAG architecture with a Mistral-based LLM and a knowledge base built from the AttackER dataset, which contains curated cyber-attack investigation and attribution reports.
• Retrieval is performed using dense vector embeddings and semantic similarity matching to retrieve relevant context chunks from the knowledge base.
• The LLM generates answers based on retrieved context, ensuring source attribution and reducing hallucination.
• The model supports three input types: knowledge-base queries, private repository queries, and web-based queries via retrieval from external sources.
• Few-shot prompting is applied to improve answer accuracy and relevance, with prompts including example queries and correct answers.
• Evaluation uses a custom QA pair dataset and metrics including faithfulness, answer relevancy, context precision, and correctness.

Experimental results

Research questions

• RQ1 Can a RAG-based QA system outperform general-purpose LLMs like GPT-3.5 and GPT-4o in providing accurate, source-credited answers for cyber-attack attribution?
• RQ2 How does few-shot prompting affect the quality and reliability of answers in a cyber-attack investigation context?
• RQ3 To what extent does the RAG model reduce hallucination compared to fine-tuned or zero-shot LLMs?
• RQ4 How effective is the system in retrieving relevant context when multiple similar contexts exist?
• RQ5 What are the limitations of the current RAG model in terms of retrieval accuracy and response latency?

Key findings

• The RAG-based QA model achieved 85% answer relevancy in zero-shot and 91% in few-shot settings, indicating strong instruction following and improved response quality with examples.
• The model outperformed GPT-3.5 and GPT-4o by providing source citations and significantly reducing hallucination, especially in out-of-distribution queries.
• GPT-3.5 exhibited notable hallucination, generating false details about CVE-2023-34362 despite initially acknowledging its knowledge cutoff.
• GPT-4o provided accurate answers but lacked transparency in source attribution and was rate-limited by the API, reducing practical utility.
• Incorrect context retrieval led to inaccurate answers in some cases, such as misidentifying the actor behind the 'Sharpshooter' campaign due to retrieval bias.
• The model demonstrated high answer similarity to ground truth, indicating consistency and reliability in response generation.