[Paper Review] Cybersecurity Threat Hunting and Vulnerability Analysis Using a Neo4j Graph Database of Open Source Intelligence

Elijah Pelofske, Lorie M. Liebrock+1 more|arXiv (Cornell University)|Jan 27, 2023

Advanced Malware Detection Techniques4 citations

TL;DR

This paper proposes a Neo4j-based graph database that aggregates open source intelligence (OSINT) from diverse sources—such as threat reports, social media, and antivirus scans—to automate the discovery and correlation of indicators of compromise (IoCs). By extracting structured IoCs (e.g., IPs, domains, hashes, CVEs) and linking them to unstructured OSINT documents via machine learning-enhanced metadata, the system enables efficient threat hunting, malware family clustering, and vulnerability analysis through semantic graph queries, significantly improving the scalability and insight extraction from noisy, large-scale OSINT.

ABSTRACT

Open source intelligence is a powerful tool for cybersecurity analysts to gather information both for analysis of discovered vulnerabilities and for detecting novel cybersecurity threats and exploits. However the scale of information that is relevant for information security on the internet is always increasing, and is intractable for analysts to parse comprehensively. Therefore methods of condensing the available open source intelligence, and automatically developing connections between disparate sources of information, is incredibly valuable. In this research, we present a system which constructs a Neo4j graph database formed by shared connections between open source intelligence text including blogs, cybersecurity bulletins, news sites, antivirus scans, social media posts (e.g., Reddit and Twitter), and threat reports. These connections are comprised of possible indicators of compromise (e.g., IP addresses, domains, hashes, email addresses, phone numbers), information on known exploits and techniques (e.g., CVEs and MITRE ATT&CK Technique ID's), and potential sources of information on cybersecurity exploits such as twitter usernames. The construction of the database of potential IoCs is detailed, including the addition of machine learning and metadata which can be used for filtering of the data for a specific domain (for example a specific natural language) when needed. Examples of utilizing the graph database for querying connections between known malicious IoCs and open source intelligence documents, including threat reports, are shown. We show three specific examples of interesting connections found in the graph database; the connections to a known exploited CVE, a known malicious IP address, and a malware hash signature.

Motivation & Objective

• Address the challenge of scaling and noise in open source intelligence (OSINT) for cybersecurity threat detection.
• Automate the identification and correlation of indicators of compromise (IoCs) from unstructured OSINT sources such as blogs, news, social media, and threat reports.
• Construct a scalable, queryable knowledge graph that links IoCs to relevant OSINT documents using semantic relationships and metadata.
• Enable efficient threat hunting, malware family clustering, and vulnerability analysis through graph-based querying of interconnected cybersecurity data.

Proposed method

• Aggregate OSINT from diverse sources including threat reports, social media (Reddit, Twitter), antivirus scan results, and news sites.
• Extract potential IoCs (e.g., IPs, domains, hashes, CVEs, MITRE ATT&CK IDs) using pattern matching from unstructured text.
• Construct a Neo4j graph database where nodes represent IoCs and OSINT documents, and edges represent co-occurrence or contextual relationships.
• Enhance data quality by integrating machine learning models to classify document relevance (cybersecurity vs. non-cybersecurity content) and detect exploit techniques.
• Incorporate metadata such as language detection and confidence scores to filter and prioritize high-quality, domain-specific OSINT documents.
• Enable interactive querying and visualization of graph relationships using Neo4j’s native interface and external tools like PyGraphistry for large-scale network exploration.

Experimental results

Research questions

• RQ1 How can a graph database model effectively integrate and correlate diverse, unstructured OSINT sources to improve threat detection?
• RQ2 To what extent can machine learning-enhanced metadata improve the relevance and quality of extracted IoCs from OSINT documents?
• RQ3 Can graph-based querying of IoC relationships enable more effective malware family clustering and infrastructure discovery than traditional methods?
• RQ4 How scalable and efficient is the system in identifying complex, multi-hop relationships between IoCs and threat documents across large OSINT corpora?

Key findings

• The system successfully constructed a large-scale Neo4j graph database containing interconnected IoCs and OSINT documents from diverse sources, enabling semantic querying of cybersecurity relationships.
• Graph queries revealed meaningful connections between known malicious IoCs and related OSINT documents, demonstrating utility in threat hunting and infrastructure mapping.
• Malware samples with identical resource files (e.g., same OriginalFilename and embedded resources) were identified as likely related, even with different hashes, indicating shared development origins.
• Visualizations of large graph expansions (e.g., Qakbot IP connections) showed that even a few degrees of separation can generate highly connected subgraphs, highlighting the value of graph-based exploration.
• The integration of CTC and language detection metadata significantly improved query precision by filtering for English-language, cybersecurity-relevant documents.
• The system demonstrated scalability and utility in identifying clusters of related malware and infrastructure, supporting effective malware family clustering and vulnerability analysis.