[论文解读] Network Traffic Anomaly Detection Using Recurrent Neural Networks

Benjamin J. Radford, Leonardo M. Apolonio+2 more|arXiv (Cornell University)|Mar 28, 2018

Network Security and Intrusion Detection参考文献 9被引用 122

一句话总结

本论文在令牌化的 netflow 序列上训练一个无监督的 LSTM RNN，以建模正常的网络行为并检测异常流量，在 ISCX IDS 数据集上的 AUC 为 0.84。

ABSTRACT

We show that a recurrent neural network is able to learn a model to represent sequences of communications between computers on a network and can be used to identify outlier network traffic. Defending computer networks is a challenging problem and is typically addressed by manually identifying known malicious actor behavior and then specifying rules to recognize such behavior in network communications. However, these rule-based approaches often generalize poorly and identify only those patterns that are already known to researchers. An alternative approach that does not rely on known malicious behavior patterns can potentially also detect previously unseen patterns. We tokenize and compress netflow into sequences of "words" that form "sentences" representative of a conversation between computers. These sentences are then used to generate a model that learns the semantic and syntactic grammar of the newly generated language. We use Long-Short-Term Memory (LSTM) cell Recurrent Neural Networks (RNN) to capture the complex relationships and nuances of this language. The language model is then used predict the communications between two IPs and the prediction error is used as a measurement of how typical or atyptical the observed communication are. By learning a model that is specific to each network, yet generalized to typical computer-to-computer traffic within and outside the network, a language model is able to identify sequences of network activity that are outliers with respect to the model. We demonstrate positive unsupervised attack identification performance (AUC 0.84) on the ISCX IDS dataset which contains seven days of network activity with normal traffic and four distinct attack patterns.

研究动机与目标

• 动机：展示用异常检测来防御网络的挑战，超越基于规则的系统。
• 提出一种基于无监督的 LSTM 的语言模型，以学习网络流量序列的语法。
• 展示学习到的模型可以在没有标注攻击数据的情况下识别异常流量。
• 显示原型字节序列特征在检测性能上优于服务端口特征。

提出的方法

• 将 netflow 进行分词并压缩为表示计算机之间对话的词序列。
• 使用两组特征：proto-byte 序列和 service-port 序列，采用 20% dropout 和两层堆叠的双向 LSTM。
• 训练一个嵌入层，将令牌映射到 100 维空间，并在一个 10-token 窗口内预测下一个令牌。
• 将异常分数评估为预测的多类别对数损失，使用每个 dyad-hour 的最大异常分数作为异常指示。
• 比较干净基线、污染基线和 NoDoS 场景，以评估鲁棒性和数据需求。

实验结果

研究问题

• RQ1 无监督的 LSTM 语言模型能否在没有标注攻击数据的情况下学习典型的网络流量并识别异常或恶意活动？
• RQ2 不同的标记方案（proto-byte 与 service port）如何影响对网络流量的异常检测性能？
• RQ3 在带有攻击数据（脏数据）进行训练是否比在干净数据上训练能提高异常检测？
• RQ4 在训练数据中移除 DoS/DDoS 流量（NoDoS）时，该方法是否仍然鲁棒？

主要发现

• Proto-byte 序列模型在所有场景中均明显优于 service-port 序列模型。
• 脏基线模型（在包含攻击的数据上训练）在所有情况下都优于干净基线模型。
• NoDoS 原型字节模型在攻击检测方面的表现与脏基线原型字节模型在实质上相同。
• 该方法在 ISCX IDS 数据集上实现了 0.84 的 AUC，用于无监督异常检测。
• 该方法表明学习一个网络特定但可泛化的模型即可在没有干净训练数据的情况下标记恶意活动。