[Paper Review] Explainable Intrusion Detection Systems (X-IDS): A Survey of Current Methods, Challenges, and Opportunities

Subash Neupane, Jesse Ables+5 more|arXiv (Cornell University)|Jul 13, 2022

Network Security and Intrusion Detection4 citations

TL;DR

This survey proposes a systematic framework for Explainable Intrusion Detection Systems (X-IDS) by analyzing white-box and black-box explainability approaches in AI-based IDS. It presents a three-layered, human-in-the-loop architecture, identifies key challenges like model opacity and adversarial attacks, and recommends standardized metrics and stakeholder-specific explanations to improve trust and decision-making in Cyber Security Operations Centers (CSoCs).

ABSTRACT

The application of Artificial Intelligence (AI) and Machine Learning (ML) to cybersecurity challenges has gained traction in industry and academia, partially as a result of widespread malware attacks on critical systems such as cloud infrastructures and government institutions. Intrusion Detection Systems (IDS), using some forms of AI, have received widespread adoption due to their ability to handle vast amounts of data with a high prediction accuracy. These systems are hosted in the organizational Cyber Security Operation Center (CSoC) as a defense tool to monitor and detect malicious network flow that would otherwise impact the Confidentiality, Integrity, and Availability (CIA). CSoC analysts rely on these systems to make decisions about the detected threats. However, IDSs designed using Deep Learning (DL) techniques are often treated as black box models and do not provide a justification for their predictions. This creates a barrier for CSoC analysts, as they are unable to improve their decisions based on the model's predictions. One solution to this problem is to design explainable IDS (X-IDS). This survey reviews the state-of-the-art in explainable AI (XAI) for IDS, its current challenges, and discusses how these challenges span to the design of an X-IDS. In particular, we discuss black box and white box approaches comprehensively. We also present the tradeoff between these approaches in terms of their performance and ability to produce explanations. Furthermore, we propose a generic architecture that considers human-in-the-loop which can be used as a guideline when designing an X-IDS. Research recommendations are given from three critical viewpoints: the need to define explainability for IDS, the need to create explanations tailored to various stakeholders, and the need to design metrics to evaluate explanations.

Motivation & Objective

• Address the lack of transparency in deep learning-based Intrusion Detection Systems (IDS), which are treated as black boxes and hinder analyst trust and decision-making.
• Systematically review current methods of explainable AI (XAI) in IDS, including white-box and post-hoc (black-box) explanation techniques.
• Identify critical challenges in X-IDS, such as adversarial attacks on explanations, misleading interpretations, and performance overhead.
• Propose a generic, three-layered X-IDS architecture aligned with DARPA’s XAI framework to support human-in-the-loop security operations.
• Provide research recommendations for defining explainability, tailoring explanations to stakeholders, and developing evaluation metrics.

Proposed method

• Categorize X-IDS methods into white-box (inherently interpretable models like attention mechanisms or decision trees) and black-box (post-hoc explainers like LIME and SHAP) approaches.
• Analyze trade-offs between model performance and explainability, noting that black-box methods often achieve higher accuracy but require additional explanation computation.
• Propose a three-layered X-IDS architecture: (1) data ingestion and preprocessing, (2) explainable ML model (e.g., explainable DNN or hybrid model), and (3) explanation interface for CSoC analysts.
• Integrate human-in-the-loop feedback by enabling analysts to validate, correct, or retrain explanations based on domain knowledge.
• Use DARPA’s XAI framework as a design guideline to ensure modularity, extensibility, and alignment with real-world CSoC workflows.
• Conduct a comprehensive survey of 100+ recent XAI and IDS papers to identify trends, gaps, and open challenges in the field.

Experimental results

Research questions

• RQ1 How do white-box and black-box explainability approaches differ in their ability to balance detection accuracy and explanation transparency in IDS?
• RQ2 What are the key challenges in deploying X-IDS in real-world Cyber Security Operations Centers (CSoCs), including adversarial attacks on explanations and performance overhead?
• RQ3 How can explanations be tailored to different stakeholders (e.g., analysts, auditors, system architects) to improve trust and decision-making?
• RQ4 What metrics are needed to objectively evaluate the quality and reliability of explanations in X-IDS?
• RQ5 How can a scalable, modular, and human-centered X-IDS architecture be designed to support real-time threat detection and explanation?

Key findings

• Black-box explanation methods like LIME and SHAP are more widely adopted than white-box models due to their compatibility with high-accuracy deep learning models.
• White-box models generally offer more faithful and detailed explanations but often sacrifice detection performance compared to black-box models.
• Adversarial attacks on explanations—such as data poisoning or model evasion—can manipulate or degrade explanation quality, posing a serious threat to X-IDS reliability.
• Misleading or incorrect explanations, whether due to poor-quality data or flawed explanation algorithms, can mislead CSoC analysts and degrade system trust.
• Performance overhead from explanation generation is a critical concern; selective explanation (explaining only high-risk samples) is a viable optimization strategy to maintain real-time operation.
• There is no consensus on the definition of 'explainability' in IDS, and no standardized metrics exist to evaluate explanation quality, highlighting a major research gap.