[Paper Review] Determining the Scale of Impact from Denial-of-Service Attacks in Real Time Using Twitter

Chi Zhang, Bryan Wilkinson+2 more|arXiv (Cornell University)|Sep 12, 2019

Network Security and Intrusion Detection27 references4 citations

TL;DR

This paper proposes a weakly-supervised NLP model that leverages Latent Dirichlet Allocation (LDA) and symmetric Kullback-Leibler divergence to detect DDoS attack-related tweets in real time using Twitter data, enabling rapid estimation of attack impact scale. The method achieves precision comparable to supervised models and generalizes across financial institutions, with optional fine-tuning via a supervised classifier layer to improve precision at the cost of recall.

ABSTRACT

Denial of Service (DoS) attacks are common in on-line and mobile services such as Twitter, Facebook and banking. As the scale and frequency of Distributed Denial of Service (DDoS) attacks increase, there is an urgent need for determining the impact of the attack. Two central challenges of the task are to get feedback from a large number of users and to get it in a timely manner. In this paper, we present a weakly-supervised model that does not need annotated data to measure the impact of DoS issues by applying Latent Dirichlet Allocation and symmetric Kullback-Leibler divergence on tweets. There is a limitation to the weakly-supervised module. It assumes that the event detected in a time window is a DoS attack event. This will become less of a problem, when more non-attack events twitter got collected and become less likely to be identified as a new event. Another way to remove that limitation, an optional classification layer, trained on manually annotated DoS attack tweets, to filter out non-attack tweets can be used to increase precision at the expense of recall. Experimental results show that we can learn weakly-supervised models that can achieve comparable precision to supervised ones and can be generalized across entities in the same industry.

Motivation & Objective

• To address the challenge of rapidly assessing the scale of impact from DDoS attacks using real-time user feedback.
• To develop a scalable, low-resource method for detecting attack-related tweets without requiring large annotated datasets.
• To enable real-time estimation of attack severity using only tweet content and temporal patterns.
• To evaluate model generalization across entities within the same industry, such as banks.
• To explore the trade-off between precision and recall through an optional supervised classification layer.

Proposed method

• The model applies Latent Dirichlet Allocation (LDA) to extract topics from historical and real-time tweet streams.
• It computes symmetric Kullback-Leibler (KL) divergence between the distribution of new topics and historical topic distributions to detect anomalies.
• An anomaly score is generated based on topic distribution shifts, identifying potential DDoS-related events.
• The number of topics is determined empirically using a hyperparameter α, with α = 10 found to yield optimal performance.
• An optional supervised classifier layer is trained on manually annotated attack tweets to filter false positives, enhancing precision at the cost of recall.
• Impact severity is estimated using a formula based on the ratio and count of detected attack-related tweets.

Experimental results

Research questions

• RQ1 Can weakly-supervised topic modeling detect DDoS attack-related tweets in real time with high precision?
• RQ2 How well does the model generalize across different financial institutions using only unannotated data?
• RQ3 What is the trade-off between precision and recall when adding a supervised classification layer?
• RQ4 How sensitive is model performance to the choice of the topic count hyperparameter α?
• RQ5 Can the model reliably detect attack events without relying on manually labeled training data?

Key findings

• The weakly-supervised model achieves precision comparable to a supervised baseline model, demonstrating strong performance without annotated training data.
• The model generalizes effectively across entities in the same domain, such as Bank of America, PNC, and Wells Fargo, indicating robustness to entity-specific language variation.
• The addition of a supervised classifier layer increases precision by filtering non-attack tweets, though it reduces recall due to stricter filtering.
• The model's performance is highly sensitive to the choice of the hyperparameter α, with α = 10 yielding optimal results in experiments.
• The Hierarchical Dirichlet Process (HDP) model for automatic topic count estimation performed poorly on tweets, likely due to the short, noisy nature of microblog content.
• Human agreement on tweet classification was low (only 2 out of 34 tweets had unanimous labeling), highlighting the difficulty of manual annotation and the value of automated detection.