[论文解读] Collective Anomaly Detection based on Long Short Term Memory Recurrent Neural Network

Loïc Bontemps, Van Loi Cao+2 more|arXiv (Cornell University)|Mar 28, 2017

Network Security and Intrusion Detection参考文献 10被引用 28

一句话总结

本文提出了一种基于长短期记忆（LSTM）循环神经网络的实时集体异常检测模型，该模型仅使用正常网络流量进行训练。通过分析多个最近时间步的预测误差，该方法能够检测出在单个时间步中不明显的集体异常——即跨多个时间步的协调性偏差，在KDD 1999数据集上实现了可靠的检测效果，并提升了对协同攻击的敏感度。

ABSTRACT

Intrusion detection for computer network systems becomes one of the most critical tasks for network administrators today. It has an important role for organizations, governments and our society due to its valuable resources on computer networks. Traditional misuse detection strategies are unable to detect new and unknown intrusion. Besides, anomaly detection in network security is aim to distinguish between illegal or malicious events and normal behavior of network systems. Anomaly detection can be considered as a classification problem where it builds models of normal network behavior, which it uses to detect new patterns that significantly deviate from the model. Most of the cur- rent research on anomaly detection is based on the learning of normally and anomaly behaviors. They do not take into account the previous, re- cent events to detect the new incoming one. In this paper, we propose a real time collective anomaly detection model based on neural network learning and feature operating. Normally a Long Short Term Memory Recurrent Neural Network (LSTM RNN) is trained only on normal data and it is capable of predicting several time steps ahead of an input. In our approach, a LSTM RNN is trained with normal time series data before performing a live prediction for each time step. Instead of considering each time step separately, the observation of prediction errors from a certain number of time steps is now proposed as a new idea for detecting collective anomalies. The prediction errors from a number of the latest time steps above a threshold will indicate a collective anomaly. The model is built on a time series version of the KDD 1999 dataset. The experiments demonstrate that it is possible to offer reliable and efficient for collective anomaly detection.

研究动机与目标

• 解决传统异常检测方法孤立分析单个时间步的局限性。
• 检测集体异常——即跨多个时间步的协调性偏差，而非孤立异常。
• 通过建模网络流量中的时间依赖性，提升对复杂且此前未见的攻击的检测能力。
• 开发一种仅使用正常数据进行训练的实时、可扩展解决方案，基于LSTM RNN。
• 证明利用预测误差累积作为集体异常检测信号的有效性。

提出的方法

• 在KDD 1999数据集的正常时间序列网络流量数据上，仅对长短期记忆（LSTM）RNN进行训练。
• 训练后的LSTM基于最近的输入序列预测未来的时间步，为每个时间步生成预测输出。
• 为每个时间步计算预测误差——即实际值与预测值之间的差异。
• 维护一个近期预测误差的滑动窗口，并持续监控其累积幅度。
• 若窗口内预测误差的总和超过预设阈值，则标记为集体异常。
• 该方法利用时间上下文和序列建模能力，检测通过单步分析无法识别的微弱且协调的偏差。

实验结果

研究问题

• RQ1 仅在正常数据上训练的LSTM RNN能否通过分析随时间变化的预测误差模式来检测集体异常？
• RQ2 与单步异常检测相比，跨多个时间步聚合预测误差如何提升对协同攻击的检测能力？
• RQ3 在真实网络流量中，对预测误差累积采用何种阈值策略可实现最佳检测性能？
• RQ4 该模型能否在未接触异常样本的情况下检测出此前未见的攻击模式？
• RQ5 该模型在KDD 1999基准数据集的实时检测场景中表现如何？

主要发现

• 该模型通过识别多个时间步内预测误差的持续偏差，成功检测出集体异常。
• 将近期多个时间步的预测误差进行聚合，显著提升了对协同攻击的检测能力，优于逐步异常检测方法。
• 该方法在KDD 1999数据集上实现了可靠的检测性能，对多种攻击模式表现出强鲁棒性。
• 该方法在训练阶段无需标注的异常数据，仅依赖正常数据，从而增强了泛化能力。
• 该模型具备实时推理能力，适用于实际网络监控系统的部署。
• 结果表明，通过LSTM预测误差分析实现的集体异常检测是一种可行且有效的入侵检测策略。