[Paper Review] Online Human-Bot Interactions: Detection, Estimation, and Characterization

Onur Varol, Emilio Ferrara+3 more|arXiv (Cornell University)|Mar 9, 2017

Spam and Phishing DetectionComputer Science236 citations

TL;DR

A framework detects Twitter bots using 1,150 features from public data, achieving high accuracy and estimating bot prevalence between 9% and 15%.

ABSTRACT

Increasing evidence suggests that a growing amount of social media content is generated by autonomous entities known as social bots. In this work we present a framework to detect such entities on Twitter. We leverage more than a thousand features extracted from public data and meta-data about users: friends, tweet content and sentiment, network patterns, and activity time series. We benchmark the classification framework by using a publicly available dataset of Twitter bots. This training data is enriched by a manually annotated collection of active Twitter users that include both humans and bots of varying sophistication. Our models yield high accuracy and agreement with each other and can detect bots of different nature. Our estimates suggest that between 9% and 15% of active Twitter accounts are bots. Characterizing ties among accounts, we observe that simple bots tend to interact with bots that exhibit more human-like behaviors. Analysis of content flows reveals retweet and mention strategies adopted by bots to interact with different target groups. Using clustering analysis, we characterize several subclasses of accounts, including spammers, self promoters, and accounts that post content from connected applications.

Motivation & Objective

• Develop a scalable framework to detect social bot accounts on Twitter using extensive public data and metadata.
• Evaluate detection accuracy across datasets, models, and evolving bot sophistication.
• Estimate the prevalence of bot-like accounts in a large English-speaking Twitter population.
• Characterize social connectivity, information flow, and behavioral clusters among human and bot-like accounts.

Proposed method

• Extract 1,150 features from user metadata, content, network structure, and timing across six feature classes.
• Train supervised machine learning classifiers (Random Forest, AdaBoost, Logistic Regression, Decision Tree) and select the best via AUC, with Random Forest achieving 0.95 AUC.
• Annotate and expand training data with a manually labeled set of bot and human accounts to test generalization.
• Evaluate models on honeypot bot data and manually annotated recent accounts, assessing cross-dataset performance and threshold selection.
• Compute bot-score thresholds by maximizing classification accuracy across deciles of manually annotated data.

Experimental results

Research questions

• RQ1 Can a large feature-driven supervised model accurately distinguish bots from humans on Twitter?
• RQ2 How does bot sophistication evolve, and how does this affect model performance across datasets?
• RQ3 What is the estimated prevalence of bot-like accounts in a large English-speaking Twitter population?
• RQ4 What are the patterns of social connectivity and information flow among human versus bot-like accounts?
• RQ5 What behavioral clusters emerge among accounts, and what characterizes each cluster?

Key findings

• A large-feature framework yields high detection performance; 0.95 AUC with Random Forest on honeypot data.
• On manually annotated data, accuracy ranges from over 90% in low bot-score deciles to 60–80% in the challenging mid-range, with overall 86% accuracy when weighted by population.
• Bot prevalence is estimated between 9% and 15% depending on training data and threshold selection.
• Humans predominantly follow humans and are followed by humans and some sophisticated bots, while bots favor bot-to-bot interactions and exhibit lower reciprocity.
• Clustering reveals ten behavioral cohorts; notable clusters include recruiters/spam accounts, accounts posting from connected apps, and mixed bot/human (cyborg) accounts.