[Paper Review] Combinatorial Optimization based Feature Selection Method: A study on Network Intrusion Detection

Anjum Nazir, Rizwan Ahmed Khan|arXiv (Cornell University)|Jun 11, 2019

Network Security and Intrusion Detection36 references4 citations

TL;DR

This paper proposes TS-RF, a novel wrapper-based feature selection method combining Tabu Search (TS) metaheuristic and Random Forest (RF) classifier to enhance network intrusion detection on the UNSW-NB15 dataset. By reducing feature count by over 60%, the method improves detection accuracy while lowering computational complexity, demonstrating strong potential for identifying unknown and advanced cyberattacks.

ABSTRACT

Advancements in computer networks and communication technologies like software defined networks (SDN), Internet of things (IoT), microservices architecture, cloud computing and network function virtualization (NFV) have opened new fronts and challenges for security experts to combat against modern cyberattacks. Relying on perimeter defense and signature-based network security solutions like Intrusion Detection and Prevention Systems (IDS/IPS) have failed to deliver adequate level of security against new attack vectors such as advance persistent threats, zero days, ransomware, botnets and other forms of targeted attacks. Recent developments in machine learning and cognitive computing have shown great potential to detect unknown and new intrusion events where legacy misuse and anomaly based intrusion detection systems usually fail. In this research study we applied state of the art machine learning algorithms on UNSW-NB15 dataset for potential applicability to detect new attacks. We also proposed a novel wrapper based feature selection technique TS-RF using metaheuristic Tabu Search (TS) algorithm and Random Forest (RF) ensemble classifier. Results obtained by applying proposed feature selection technique i.e. TS-RF on UNSW-NB15 dataset show improvement in overall intrusion detection accuracy while it reduces computation complexity as it removes more than 60% features.

Motivation & Objective

• To address the limitations of traditional signature-based and anomaly-based intrusion detection systems in detecting advanced threats like zero-day attacks and APTs.
• To improve detection accuracy for unknown and emerging cyber threats using machine learning on the UNSW-NB15 dataset.
• To reduce computational complexity in intrusion detection by eliminating redundant and irrelevant features through an intelligent feature selection approach.
• To develop and evaluate a novel wrapper-based feature selection technique that integrates metaheuristic optimization with ensemble learning.

Proposed method

• The proposed method, TS-RF, uses the Tabu Search (TS) metaheuristic algorithm to explore the feature subset space and identify optimal combinations of features.
• The Random Forest (RF) classifier serves as the evaluation function within the wrapper framework, providing accurate performance estimation for each feature subset.
• Feature selection is performed iteratively by TS, which maintains a tabu list to avoid revisiting previously explored solutions and escape local optima.
• The algorithm evaluates each candidate feature subset using RF classification accuracy, guiding the search toward high-performing, compact feature sets.
• The process continues until a stopping criterion is met, such as a maximum number of iterations or no improvement in solution quality.
• The final selected feature subset is used to train the RF model, resulting in a more efficient and accurate intrusion detection system.

Experimental results

Research questions

• RQ1 Can a hybrid metaheuristic-wrapper approach improve intrusion detection accuracy on the UNSW-NB15 dataset compared to baseline methods?
• RQ2 To what extent can feature selection reduce computational complexity without degrading detection performance?
• RQ3 How effective is the integration of Tabu Search with Random Forest in identifying optimal feature subsets for intrusion detection?
• RQ4 Can the proposed TS-RF method detect previously unseen or zero-day attack patterns more effectively than traditional IDS/IPS systems?

Key findings

• The TS-RF method achieved improved overall intrusion detection accuracy compared to baseline models using the full feature set.
• The proposed technique successfully reduced the number of features by more than 60% while maintaining or enhancing detection performance.
• The use of Tabu Search enabled effective exploration of the feature space, avoiding local optima and identifying high-quality subsets.
• The integration of Random Forest as the evaluation function contributed to robust and reliable performance estimation during feature selection.
• The method demonstrated strong potential for detecting unknown and advanced cyberattacks, including zero-day and APT-style threats.
• The computational complexity of the intrusion detection system was significantly reduced due to the substantial reduction in feature dimensionality.