[Paper Review] An AI-based, Multi-stage detection system of banking botnets

Ling Li, Zhiqiang Gao+3 more|arXiv (Cornell University)|Jul 18, 2019

Spam and Phishing Detection6 references4 citations

TL;DR

This paper presents an AI-driven, multi-stage detection system for banking botnets that leverages Cyber Data Lakes and deep learning models—specifically LSTM networks—for early warning, DGA detection, phishing URL identification, and DNS tunneling. The system achieves a 10x reduction in false positives compared to baseline models, with AUC scores of 0.9965 for DGA and 0.9956 for phishing URL classification.

ABSTRACT

Banking Trojans, botnets are primary drivers of financially-motivated cybercrime. In this paper, we first analyzed how an APT-based banking botnet works step by step through the whole lifecycle. Specifically, we present a multi-stage system that detects malicious banking botnet activities which potentially target the organizations. The system leverages Cyber Data Lake as well as multiple artificial intelligence techniques at different stages. The evaluation results using public datasets showed that Deep Learning based detections were highly successful compared with baseline models.

Motivation & Objective

• Address the growing threat of APT-based banking botnets that evade traditional detection and extend dwell times up to 498 days.
• Reduce false positive rates in botnet detection, which is critical for high-volume network environments.
• Enable early detection of spear phishing campaigns and malicious infrastructure before full compromise occurs.
• Integrate multiple AI techniques—LSTM, graph analysis—into a unified detection framework using centralized Cyber Data Lakes.
• Improve detection accuracy across multiple stages of the cyber kill chain using deep learning on public and internal threat data.

Proposed method

• Utilizes a Cyber Data Lake as a centralized platform to ingest, normalize, and store diverse security datasets from internal and external sources.
• Employs a multi-stage detection pipeline aligned with the Cyber Kill Chain (CKC) model, covering reconnaissance, delivery, exploitation, C2, and action on objectives.
• Applies Long Short-Term Memory (LSTM) networks to classify domains as malicious (DGA-generated) or benign using character-level sequences of domain names.
• Uses LSTM-based models for phishing URL detection with input sequences up to 128 characters, leveraging character embeddings and dropout (0.5) for regularization.
• Implements early stopping and hyperparameter tuning during training to prevent overfitting, with model evaluation using ROC curves and AUC.
• Applies whitelists and blacklists in the operational phase to reduce noise and improve detection precision.

Experimental results

Research questions

• RQ1 Can deep learning models like LSTM significantly reduce false positive rates in DGA and phishing URL detection compared to traditional n-gram and bag-of-words baselines?
• RQ2 To what extent can a multi-stage detection system based on Cyber Data Lakes detect botnet activities across the entire attack lifecycle, including early-stage reconnaissance?
• RQ3 How effective are LSTM-based models in identifying malicious domains and URLs when trained on public datasets and evaluated using AUC and FPR metrics?
• RQ4 Can the integration of AI techniques such as LSTM and graph analysis improve detection accuracy and reduce dwell time in banking botnet campaigns?
• RQ5 What is the performance trade-off between detection rate and false positive rate when using advanced deep learning models versus conventional machine learning models?

Key findings

• The LSTM model achieved an AUC of 0.9965 for DGA classification, significantly outperforming the baseline n-gram model (AUC 0.9610).
• For phishing URL detection, the LSTM model reached an AUC of 0.9956, compared to 0.9687 for the baseline bag-of-words model.
• At a true positive rate of 91%, the LSTM model achieved a false positive rate of only 0.7%, while the baseline model had a false positive rate of 8% under the same detection threshold.
• The LSTM model reduced false positives by over 10x compared to the baseline model at equivalent detection rates, demonstrating strong practical value in high-volume environments.
• Training history analysis confirmed that the LSTM models did not overfit, as indicated by stable and consistent validation performance across epochs.
• The system demonstrated high efficiency with a 6x speedup using hardware-optimized LSTM inference on a GPU cluster, enabling scalable deployment in production settings.