[Paper Review] SIGMA : Strengthening IDS with GAN and Metaheuristics Attacks

Simon Msika, Alejandro Quintero+1 more|arXiv (Cornell University)|Dec 18, 2019

Advanced Malware Detection Techniques24 references4 citations

TL;DR

SIGMA proposes an iterative adversarial training method that combines Generative Adversarial Networks (GANs) and a hybrid metaheuristic algorithm (local-search and genetic algorithm) to generate realistic adversarial attacks for retraining machine learning-based Intrusion Detection Systems (IDS). The method significantly improves IDS robustness, achieving up to 100% detection rate on generated attacks in as few as two training rounds, especially when combining GAN and metaheuristic-generated data.

ABSTRACT

An Intrusion Detection System (IDS) is a key cybersecurity tool for network administrators as it identifies malicious traffic and cyberattacks. With the recent successes of machine learning techniques such as deep learning, more and more IDS are now using machine learning algorithms to detect attacks faster. However, these systems lack robustness when facing previously unseen types of attacks. With the increasing number of new attacks, especially against Internet of Things devices, having a robust IDS able to spot unusual and new attacks becomes necessary. This work explores the possibility of leveraging generative adversarial models to improve the robustness of machine learning based IDS. More specifically, we propose a new method named SIGMA, that leverages adversarial examples to strengthen IDS against new types of attacks. Using Generative Adversarial Networks (GAN) and metaheuristics, SIGMA %Our method consists in generates adversarial examples, iteratively, and uses it to retrain a machine learning-based IDS, until a convergence of the detection rate (i.e. until the detection system is not improving anymore). A round of improvement consists of a generative phase, in which we use GANs and metaheuristics to generate instances ; an evaluation phase in which we calculate the detection rate of those newly generated attacks ; and a training phase, in which we train the IDS with those attacks. We have evaluated the SIGMA method for four standard machine learning classification algorithms acting as IDS, with a combination of GAN and a hybrid local-search and genetic algorithm, to generate new datasets of attacks. Our results show that SIGMA can successfully generate adversarial attacks against different machine learning based IDS. Also, using SIGMA, we can improve the performance of an IDS to up to 100\% after as little as two rounds of improvement.

Motivation & Objective

• Address the vulnerability of machine learning-based IDS to previously unseen adversarial attacks, especially in IoT environments with rising cyber threats.
• Overcome the fragility of deep learning models in IDS by proactively training them on adversarially generated attacks.
• Improve the robustness and generalization capability of IDS against novel, unseen attack patterns using iterative adversarial training.
• Investigate whether combining GAN-generated attacks with metaheuristic-optimized attacks enhances IDS resilience more effectively than either method alone.

Proposed method

• Use a Generative Adversarial Network (GAN) to generate realistic adversarial examples that mimic legitimate network traffic but are malicious.
• Employ a hybrid metaheuristic algorithm (local-search and genetic algorithm) to refine and optimize the generated adversarial examples for higher evasion potential.
• Iteratively apply a three-phase process: (1) generate new adversarial attack instances using GAN and metaheuristics, (2) evaluate the current IDS’s detection rate on these new attacks, and (3) retrain the IDS with the newly generated attack data.
• Continue the cycle until the detection rate converges—defined as three consecutive rounds without improvement—indicating the IDS has stabilized and generalized to the attack patterns.
• Train four different IDS classifiers (Neural Network, Random Forest, SVM, Naive Bayes) using the combined adversarial dataset to assess generalizability.
• Compare the performance of IDS trained with: (a) only GAN-generated data, (b) only metaheuristic-generated data, and (c) a combination of both, to isolate the contribution of each component.

Experimental results

Research questions

• RQ1 Can GAN-generated adversarial examples effectively evade detection by machine learning-based IDS, and to what extent?
• RQ2 Does combining GAN-generated attacks with metaheuristic-optimized attacks improve the robustness of IDS more than using either method in isolation?
• RQ3 How quickly can the SIGMA method converge to a high detection rate (e.g., 100%) on adversarial attacks across different IDS architectures?
• RQ4 To what extent does the inclusion of metaheuristic-optimized attacks enhance the generalization of IDS to unseen adversarial patterns compared to GAN-only training?
• RQ5 Is the iterative retraining process using adversarial examples effective in preventing overfitting to previously seen attack patterns?

Key findings

• The SIGMA method successfully generated adversarial attacks that evaded detection in baseline IDS models, reducing detection rates to near 0% when trained only on real data.
• For IDS based on Neural Networks and Random Forests, the SIGMA method achieved a 100% detection rate on generated attacks in as few as two rounds of retraining, significantly faster than models trained only with GAN-generated data.
• Models trained solely with metaheuristic-generated attacks failed to detect GAN-generated attacks, indicating that metaheuristics alone are insufficient to improve robustness.
• The combination of GAN and metaheuristic-generated data led to faster convergence and higher detection performance than either method used in isolation, demonstrating synergistic benefits.
• The iterative retraining process using SIGMA prevented overfitting to specific adversarial patterns and improved the IDS’s ability to generalize to new, unseen attack variants.
• The method demonstrated generalizability across four different IDS classifiers, though the most significant improvements were observed in Neural Network and Random Forest models.