[Paper Review] Network Traffic Anomaly Detection Using Recurrent Neural Networks

Benjamin J. Radford, Leonardo M. Apolonio+2 more|arXiv (Cornell University)|Mar 28, 2018

Network Security and Intrusion Detection9 references122 citations

TL;DR

The paper trains an unsupervised LSTM RNN on tokenized netflow sequences to model normal network behavior and detect anomalous traffic, achieving an AUC of 0.84 on the ISCX IDS dataset.

ABSTRACT

We show that a recurrent neural network is able to learn a model to represent sequences of communications between computers on a network and can be used to identify outlier network traffic. Defending computer networks is a challenging problem and is typically addressed by manually identifying known malicious actor behavior and then specifying rules to recognize such behavior in network communications. However, these rule-based approaches often generalize poorly and identify only those patterns that are already known to researchers. An alternative approach that does not rely on known malicious behavior patterns can potentially also detect previously unseen patterns. We tokenize and compress netflow into sequences of "words" that form "sentences" representative of a conversation between computers. These sentences are then used to generate a model that learns the semantic and syntactic grammar of the newly generated language. We use Long-Short-Term Memory (LSTM) cell Recurrent Neural Networks (RNN) to capture the complex relationships and nuances of this language. The language model is then used predict the communications between two IPs and the prediction error is used as a measurement of how typical or atyptical the observed communication are. By learning a model that is specific to each network, yet generalized to typical computer-to-computer traffic within and outside the network, a language model is able to identify sequences of network activity that are outliers with respect to the model. We demonstrate positive unsupervised attack identification performance (AUC 0.84) on the ISCX IDS dataset which contains seven days of network activity with normal traffic and four distinct attack patterns.

Motivation & Objective

• Motivate the challenge of defending networks with anomaly detection beyond rule-based systems.
• Propose an unsupervised LSTM-based language model to learn the grammar of network traffic sequences.
• Demonstrate that learned models can identify anomalous traffic without labeled attack data.
• Show that proto-byte sequence features outperform service port features in detection performance.

Proposed method

• Tokenize and compress netflow into sequences of words forming sentences representing conversations between computers.
• Use two feature sets: proto-byte sequences and service-port sequences, with a 20% dropout and two stacked bidirectional LSTM layers.
• Train an embedding layer to map tokens into a 100-dimensional space and predict the next token in a 10-token window.
• Evaluate anomaly scores as multiclass log loss on predictions, using the maximum outlier score per dyad-hour as the anomaly indicator.
• Compare clean baseline, dirty baseline, and NoDoS scenarios to assess robustness and data requirements.

Experimental results

Research questions

• RQ1 Can an unsupervised LSTM language model learn typical network traffic to identify anomalous or malicious activity without labeled attack data?
• RQ2 How do different tokenization schemes (proto-byte vs. service port) affect anomaly detection performance on network traffic?
• RQ3 Does training on dirty (attack-include) data improve anomaly detection compared to training on clean data?
• RQ4 Is the approach robust when DoS/DDoS traffic is removed (NoDoS) from training data?

Key findings

• Proto-byte sequence models consistently outperform service-port sequence models across scenarios.
• Dirty baseline models (trained on data including attacks) outperform clean baseline models in all cases.
• The NoDoS proto-byte model performs substantively the same as the dirty baseline proto-byte model for attack detection.
• The method achieves an Area Under the ROC Curve (AUC) of 0.84 on the ISCX IDS dataset for unsupervised anomaly detection.
• The approach demonstrates that learning a network-specific yet generalizable model can flag malicious activity without pristine training data.