[Paper Review] A Taxonomy of Network Threats and the Effect of Current Datasets on Intrusion Detection Systems

Hanan Hindy, David Brosset+5 more|Middlesex University Research Repository (Middlesex University Of London)|Jun 9, 2018

Network Security and Intrusion DetectionComputer Science79 references99 citations

TL;DR

The paper surveys IDS datasets and presents a threat taxonomy, finding that current IDS cover only a minority of threats and datasets lack real-world threat representations, hindering detection performance.

ABSTRACT

As the world moves towards being increasingly dependent on computers and automation, building secure applications, systems and networks are some of the main challenges faced in the current decade. The number of threats that individuals and businesses face is rising exponentially due to the increasing complexity of networks and services of modern networks. To alleviate the impact of these threats, researchers have proposed numerous solutions for anomaly detection; however, current tools often fail to adapt to ever-changing architectures, associated threats and zero-day attacks. This manuscript aims to pinpoint research gaps and shortcomings of current datasets, their impact on building Network Intrusion Detection Systems (NIDS) and the growing number of sophisticated threats. To this end, this manuscript provides researchers with two key pieces of information; a survey of prominent datasets, analyzing their use and impact on the development of the past decade's Intrusion Detection Systems (IDS) and a taxonomy of network threats and associated tools to carry out these attacks. The manuscript highlights that current IDS research covers only 33.3% of our threat taxonomy. Current datasets demonstrate a clear lack of real-network threats, attack representation and include a large number of deprecated threats, which together limit the detection accuracy of current machine learning IDS approaches. The unique combination of the taxonomy and the analysis of the datasets provided in this manuscript aims to improve the creation of datasets and the collection of real-world data. As a result, this will improve the efficiency of the next generation IDS and reflect network threats more accurately within new datasets.

Motivation & Objective

• Evaluate the limitations of available network-based datasets and their impact on IDS development.
• Review the last decade of NIDS research and its evaluation practices.
• Present a threat taxonomy categorized by source, OSI layer, and activity mode to guide dataset creation.
• Map current threats to their associated tools to help researchers build more representative datasets.

Proposed method

• Survey prominent IDS datasets and analyze their use and impact on IDS development over the past decade.
• Review recent ML/NIDS research to identify algorithmic trends and dataset dependencies.
• Construct a threat taxonomy aligned with OSI layers and active vs. passive threat characteristics.
• Map threats to their attack tools to support dataset construction and benchmarking.
• Discuss dataset generation standards and criteria to improve realism and reusability.

Experimental results

Research questions

• RQ1 What are the main limitations of available network-based datasets for IDS development?
• RQ2 To what extent do current datasets reflect real-world network threats and zero-day attacks?
• RQ3 How do recent IDS approaches perform given the gaps in datasets and threat coverage?
• RQ4 How can threat taxonomies and tool mappings guide the creation of more representative datasets?

Key findings

• Current IDS research covers only about 33.3% of the proposed threat taxonomy.
• Current datasets lack real-network threats, representation of attacks, and include many deprecated threats.
• These dataset shortcomings limit detection accuracy of contemporary ML-based IDS."
• Researchers should develop extendable, standardized dataset generation platforms to cope with concept drift in network patterns.
• The analysis highlights the dominance of certain datasets (e.g., KDD-99, DARPA) and the need for updated benchmarks reflecting modern threats.
• Mapping threats to OSI layers and associated tools can guide dataset creation and improve IDS benchmarking.