[论文解读] One-class Collective Anomaly Detection based on Long Short-Term Memory Recurrent Neural Networks

Nga Nguyen Thi, Van Loi Cao+1 more|arXiv (Cornell University)|Jan 31, 2018

Network Security and Intrusion Detection参考文献 11被引用 6

一句话总结

本文提出一种基于长短期记忆（LSTM）循环神经网络的一类集体异常检测模型，仅使用正常时间序列网络数据进行训练。通过分析多个连续时间步的预测误差，当这些误差超过阈值时检测到集体异常，在KDD 1999时间序列数据集上实现了有效的检测效果。

ABSTRACT

Intrusion detection for computer network systems has been becoming one of the most critical tasks for network administrators today. It has an important role for organizations, governments and our society due to the valuable resources hosted on computer networks. Traditional misuse detection strategies are unable to detect new and unknown intrusion types. In contrast, anomaly detection in network security aims to distinguish between illegal or malicious events and normal behavior of network systems. Anomaly detection can be considered as a classification problem where it builds models of normal network behavior, of which it uses to detect new patterns that significantly deviate from the model. Most of the current approaches on anomaly detection is based on the learning of normal behavior and anomalous actions. They do not include memory that is they do not take into account previous events classify new ones. In this paper, we propose a one class collective anomaly detection model based on neural network learning. Normally a Long Short Term Memory Recurrent Neural Network (LSTM RNN) is trained only on normal data, and it is capable of predicting several time steps ahead of an input. In our approach, a LSTM RNN is trained on normal time series data before performing a prediction for each time step. Instead of considering each time-step separately, the observation of prediction errors from a certain number of time-steps is now proposed as a new idea for detecting collective anomalies. The prediction errors of a certain number of the latest time-steps above a threshold will indicate a collective anomaly. The model is evaluated on a time series version of the KDD 1999 dataset. The experiments demonstrate that the proposed model is capable to detect collective anomaly efficiently

研究动机与目标

• 解决传统异常检测方法无法建模网络流量中时间依赖性的局限性。
• 开发一种一类学习方法，无需依赖标注的异常数据即可检测集体异常。
• 利用LSTM RNN建模正常网络行为中的序列依赖关系，以提升对复杂、多步异常的检测能力。
• 将异常识别为跨多个时间步的偏差模式，而非孤立事件。

提出的方法

• 使用仅包含正常网络时间序列数据的长短期记忆（LSTM）循环神经网络，学习时间模式。
• 基于序列输入预测未来时间步，为每个时间步生成预测误差。
• 将最近时间步滑动窗口内的预测误差聚合，形成集体异常得分。
• 当多个时间步的累积预测误差超过预设阈值时，标记为集体异常。
• 该方法专注于检测随时间协调偏离所引发的异常，而非孤立的离群点。
• 使用KDD 1999入侵检测数据集的时间序列转换版本对模型进行评估。

实验结果

研究问题

• RQ1 基于LSTM的一类模型能否通过利用时间依赖性，有效检测网络流量中的集体异常？
• RQ2 与单步异常检测相比，通过在多个时间步上累积预测误差，如何提升对复杂、多步异常的检测能力？
• RQ3 仅使用正常数据训练的模型，在多大程度上能够检测到此前未见过的集体攻击模式？
• RQ4 对预测误差采用何种阈值策略，可实现集体异常检测的最佳性能？

主要发现

• 所提出的模型通过分析时间序列中累积的预测误差，成功检测到集体异常。
• 该方法通过捕捉复杂且时间跨度较长的攻击模式，优于传统单步异常检测。
• 该模型在KDD 1999数据集的时间序列版本上实现了有效检测，展示了在类似真实环境条件下的鲁棒性。
• 该方法即使在缺乏标注异常样本的情况下依然有效，完全依赖正常训练数据。