[Paper Review] Captcha Attack: Turning Captchas Against Humanity

Mauro Conti, Luca Pajola+1 more|arXiv (Cornell University)|Jan 11, 2022

User Authentication and Security Systems4 citations

TL;DR

This paper introduces CAPA, a novel adversarial attack that exploits custom textual CAPTCHAs to evade Automatic Content Moderators (ACMs) in online social networks. By generating obfuscated text resembling classic CAPTCHAs—using typos, stylized fonts, and complex backgrounds—the attack evades OCR and NLP-based detection, achieving up to 100% evasion success on real-world ACM systems, highlighting critical vulnerabilities in current content moderation pipelines.

ABSTRACT

Nowadays, people generate and share massive content on online platforms (e.g., social networks, blogs). In 2021, the 1.9 billion daily active Facebook users posted around 150 thousand photos every minute. Content moderators constantly monitor these online platforms to prevent the spreading of inappropriate content (e.g., hate speech, nudity images). Based on deep learning (DL) advances, Automatic Content Moderators (ACM) help human moderators handle high data volume. Despite their advantages, attackers can exploit weaknesses of DL components (e.g., preprocessing, model) to affect their performance. Therefore, an attacker can leverage such techniques to spread inappropriate content by evading ACM. In this work, we propose CAPtcha Attack (CAPA), an adversarial technique that allows users to spread inappropriate text online by evading ACM controls. CAPA, by generating custom textual CAPTCHAs, exploits ACM's careless design implementations and internal procedures vulnerabilities. We test our attack on real-world ACM, and the results confirm the ferocity of our simple yet effective attack, reaching up to a 100% evasion success in most cases. At the same time, we demonstrate the difficulties in designing CAPA mitigations, opening new challenges in CAPTCHAs research area.

Motivation & Objective

• To investigate how users in online social networks evade Automatic Content Moderators (ACMs) using obfuscation techniques resembling textual CAPTCHAs.
• To reverse-engineer the concept of CAPTCHAs by showing how human-generated obfuscations can be weaponized to bypass ACMs.
• To propose CAPA, a practical adversarial attack that leverages textual CAPTCHA-like obfuscations to evade ACM detection with minimal system knowledge.
• To evaluate the feasibility and effectiveness of outlier detection as a defense mechanism against such attacks.
• To identify open challenges in distinguishing benign obfuscation from malicious evasion and to call for new detection and sanitization tools.

Proposed method

• The authors collected real-world examples of obfuscated content from social media platforms like Instagram, Facebook, and Twitter to build a taxonomy of obfuscation techniques, including leet speak, stylized characters, and background noise.
• They designed CAPA, an attack framework that generates custom textual CAPTCHAs by applying these obfuscation patterns to malicious content, making it visually human-readable but machine-ambiguous.
• The attack exploits weaknesses in OCR and NLP components of ACMs, particularly by targeting the preprocessing and inference stages where obfuscation disrupts feature extraction.
• They evaluated CAPA on real-world ACM systems, demonstrating evasion rates up to 100% across multiple platforms.
• For defense, they applied outlier detection using dimensionality reduction (PCA) followed by algorithms like Isolation Forest, LOF, ECOD, and One-Class SVM to detect anomalous, CAPTCHA-like content.
• They tuned models using grid search on contamination levels and hyperparameters, and evaluated performance using F1-score on datasets from Pinterest, Twitter, and Yahoo-Flickr.

Experimental results

Research questions

• RQ1 How do users in online social networks currently evade Automatic Content Moderators using obfuscation techniques similar to textual CAPTCHAs?
• RQ2 To what extent can custom-generated textual CAPTCHAs bypass state-of-the-art ACM systems without requiring knowledge of the target model?
• RQ3 Can outlier detection techniques effectively identify CAPTCHA-like obfuscated content in real-world social media data?
• RQ4 What is the generalization capability of detection models when faced with unknown or unseen obfuscation styles?
• RQ5 What are the fundamental challenges in distinguishing benign obfuscation from malicious evasion in content moderation?

Key findings

• CAPA achieved up to 100% evasion success against real-world ACM systems, demonstrating the high practical feasibility of the attack.
• The Local Outlier Factor (LOF) algorithm achieved an average F1-score of 80% in detecting obfuscated CAPTCHA-like content across three social media platforms.
• The performance of outlier detection stabilized with as few as four known obfuscation styles, indicating robustness to style generalization.
• False positives were estimated at around 9% of benign posts on average, with the majority stemming from text-over-background or textual content in images.
• Textual content in images—especially with hard backgrounds—was a major source of false positives, suggesting limitations in current detection heuristics.
• The study reveals that existing ACMs are vulnerable not only to model-level evasion but also to input-level obfuscation that exploits OCR and NLP pipeline weaknesses.