[Paper Review] BotGraph: Web Bot Detection Based on Sitemap

Yang Luo, Guozhen She+2 more|arXiv (Cornell University)|Mar 19, 2019

Network Security and Intrusion Detection8 references4 citations

TL;DR

BotGraph proposes a behavior-based bot detection system that models web traffic as 2D images derived from sitemap-based session subgraphs, enabling high-accuracy detection via convolutional neural networks (CNNs). It achieves ~95% precision and recall on real-world production data, effectively identifying bots without relying on easily forged identity features like UserAgent or IP addresses.

ABSTRACT

The web bots have been blamed for consuming large amount of Internet traffic and undermining the interest of the scraped sites for years. Traditional bot detection studies focus mainly on signature-based solution, but advanced bots usually forge their identities to bypass such detection. With increasing cloud migration, cloud providers provide new opportunities for an effective bot detection based on big data to solve this issue. In this paper, we present a behavior-based bot detection scheme called BotGraph that combines sitemap and convolutional neural network (CNN) to detect inner behavior of bots. Experimental results show that BotGraph achieves ~95% recall and precision on 35-day production data traces from different customers including the Bing search engine and several sites.

Motivation & Objective

• To address the challenge of detecting advanced web bots that evade traditional identity-based detection methods such as IP blocking or UserAgent filtering.
• To develop a universal, scalable bot detection scheme suitable for cloud providers managing millions of heterogeneous websites.
• To leverage behavioral patterns in web traffic—specifically, URL access sequences and frequencies—as a robust, identity-agnostic detection signal.
• To improve detection accuracy in high-bot-traffic environments where traditional methods fail due to bot fingerprint spoofing.
• To demonstrate that graph-based session modeling combined with deep learning can effectively distinguish bots from legitimate users.

Proposed method

• Construct a sitemap for each website using three methods: static file parsing, dynamic crawling, and heuristic-based inference from access logs.
• Map each client session into a subgraph of the sitemap, where nodes represent URL patterns and edges represent access transitions with associated frequencies.
• Transform each session subgraph into a 2D image representation using spatial encoding of URL access patterns and frequencies.
• Train a convolutional neural network (CNN) on these 2D images to classify sessions as bot or non-bot, using supervised learning on labeled traffic traces.
• Use the CNN’s feature maps to learn discriminative patterns in session behavior, avoiding reliance on client-side signals or identity metadata.
• Optimize inference performance to ensure low-latency deployment in production environments, with benchmarking on GPU hardware.

Experimental results

Research questions

• RQ1 Can behavior-based detection using sitemap-derived session graphs outperform identity-based bot detection in real-world, high-traffic environments?
• RQ2 To what extent can a CNN model trained on 2D images of session subgraphs achieve high precision and recall in bot detection without using IP addresses or UserAgent strings?
• RQ3 How effective is the sitemap-based graph representation in capturing distinguishing behavioral patterns between bots and legitimate users across diverse websites?
• RQ4 What is the performance of the BotGraph model under varying bot-to-user session ratios, especially in low-bot-traffic scenarios?
• RQ5 Can the model generalize across different industries and website types without retraining, given its reliance on structural session patterns rather than site-specific features?

Key findings

• BotGraph achieves approximately 95% precision and recall on 35-day production data traces from diverse sources, including Bing and multiple customer websites.
• For datasets with bot session ratio (BoS) ≥ 49%, BotGraph maintains >95% precision and recall, indicating strong performance in high-bot environments.
• Even in low-bot-traffic scenarios (BoS < 10%), BotGraph achieves >95% precision with ~70% recall, demonstrating robustness despite scattered bot behavior.
• The model outperforms traditional ML methods like SVM, XGBoost, and LSTM, especially when identity-based features (e.g., ClientIP, UserAgent) are excluded.
• False positives and false negatives are primarily observed in sessions with fewer than three URL accesses, where bot and human behavior patterns converge due to minimal interaction.
• The system introduces minimal latency on standard GPU hardware, with inference performance comparable to typical internet round-trip delays (50–100 ms).