[Paper Review] An Unsupervised Approach to Detect Spam Campaigns that Use Botnets on Twitter

Zhouhan Chen, Devika Subramanian|arXiv (Cornell University)|Apr 14, 2018

Spam and Phishing Detection8 references43 citations

TL;DR

An unsupervised, real-time system detects Twitter spam campaigns driven by botnets that tweet duplicate content with embedded URLs, builds a malware/botnet database, and provides a public API for filtering datasets.

ABSTRACT

In recent years, Twitter has seen a proliferation of automated accounts or bots that send spam, offer clickbait, compromise security using malware, and attempt to skew public opinion. Previous research estimates that around 9% to 17% of Twitter accounts are bots contributing to between 16% to 56% of tweets on the medium. This paper introduces an unsupervised approach to detect Twitter spam campaigns in real-time. The bot groups we detect tweet duplicate content with shortened embedded URLs over extended periods of time. Our experiments with the detection protocol reveal that bots consistently account for 10% to 50% of tweets generated from 7 popular URL shortening services on Twitter. More importantly, we discover that bots using shortened URLs are connected to large scale spam campaigns that control thousands of domains. There appear to be two distinct mechanisms used to control bot groups and we investigate both in this paper. Our detection system runs 24/7 and actively collects bots involved in spam campaigns and adds them to an evolving database of malicious bots. We make our database of detected bots available for query through a REST API so others can filter tweets from malicious bots to get high quality Twitter datasets for analysis.

Motivation & Objective

• Motivate and address the prevalence of Twitter bots that spread spam and malware through URL-containing tweets.
• Develop a fully unsupervised, real-time detection system that identifies bot groups and spam campaigns without labeled data.
• Enable continuous collection of malicious bots and dissemination of threat intelligence via a public API.

Proposed method

• Real-time abnormal trend monitoring of tweets containing URLs to identify trending network locations (netlocs) with a top-k approach.
• A job scheduler that gathers tweets from the most popular URL shortening services and triggers botnet detection on new URLs.
• A four-part bot detector consisting of a crawler, duplicate filter, overlap-based bot identification, and unsupervised clustering to form bot groups with high text similarity.
• A URL resolver that uses Selenium to characterize malicious behavior such as phishing or secret URLs embedded in redirects.
• A spam campaign classifier that links malicious URLs to registrant emails via WHOIS to map registrants to botnets and identify large-scale campaigns.

Experimental results

Research questions

• RQ1 Can an unsupervised, real-time system detect coordinated Twitter spam campaigns without labeled training data?
• RQ2 Do bot groups tweeting duplicated content with shortened URLs correlate with large-scale spam campaigns controlled by registrants?
• RQ3 What are the mechanisms by which spammers control Twitter botnets, and how can these be detected earlier than existing methods?
• RQ4 How effective is mapping domain registration information to botnets for identifying large-scale campaigns?

Key findings

• Bots tweet duplicate content with embedded URLs, and accounts from URL shortening services account for 10% to 50% of tweets over time.
• From Sept 10, 2017 to Nov 14, 2017, 848 unique suspicious registration emails behind 11,048 botnets with 185,922 accounts were identified.
• The system reveals large-scale campaigns that register thousands of domains and that use proxy URLs and phishing sites.
• Two bot-control mechanisms are observed: traditional botnets (batch domain/account creation) and hijacked legitimate accounts via Twitter apps.
• Three large-scale campaigns demonstrated significant activity, with registrant names linked to thousands of domains and high bot proportions in groups.