[Paper Review] Gotta CAPTCHA 'Em All: A Survey of Twenty years of the Human-or-Computer Dilemma

Meriem Guerar, Luca Verderame+3 more|arXiv (Cornell University)|Mar 2, 2021

User Authentication and Security Systems96 references4 citations

TL;DR

This paper presents a comprehensive, up-to-date survey of CAPTCHA schemes from 2000 to 2020, introducing a novel 10-category classification that includes traditional (text, image, audio) and emerging types (behavior-based, sensor-based, liveliness detection). It analyzes attack methods, evaluates security and usability trade-offs, and identifies open challenges for next-generation CAPTCHAs.

ABSTRACT

A recent study has found that malicious bots generated nearly a quarter of overall website traffic in 2019 [100]. These malicious bots perform activities such as price and content scraping, account creation and takeover, credit card fraud, denial of service, etc. Thus, they represent a serious threat to all businesses in general, but are especially troublesome for e-commerce, travel and financial services. One of the most common defense mechanisms against bots abusing online services is the introduction of Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA), so it is extremely important to understand which CAPTCHA schemes have been designed and their actual effectiveness against the ever-evolving bots. To this end, this work provides an overview of the current state-of-the-art in the field of CAPTCHA schemes and defines a new classification that includes all the emerging schemes. In addition, for each identified CAPTCHA category, the most successful attack methods are summarized by also describing how CAPTCHA schemes evolved to resist bot attacks, and discussing the limitations of different CAPTCHA schemes from the security, usability and compatibility point of view. Finally, an assessment of the open issues, challenges, and opportunities for further study is provided, paving the road toward the design of the next-generation secure and user-friendly CAPTCHA schemes.

Motivation & Objective

• To provide a comprehensive, up-to-date survey of CAPTCHA schemes from 2000 to 2020, covering both traditional and emerging types.
• To introduce a novel 10-category classification system that includes behavior-based, sensor-based, and liveliness detection CAPTCHAs, which are underrepresented in prior surveys.
• To analyze the security vulnerabilities of each CAPTCHA category by summarizing successful attack methods, including AI-based and side-channel attacks.
• To evaluate the usability, privacy, and compatibility trade-offs of modern CAPTCHA schemes, especially behavioral and sensor-based ones.
• To identify open research challenges and opportunities for designing next-generation secure, user-friendly, and privacy-preserving CAPTCHA systems.

Proposed method

• Proposed a new 10-category classification of CAPTCHA schemes: Text-based, Image-based, Audio-based, Video-based, Game-based, Slider-based, Math-based, Behavior-based, Sensor-based, and CAPTCHA for liveliness detection.
• Conducted a systematic literature review of 77 CAPTCHA schemes, compiling a timeline of their creation and breaking years with breaking percentages.
• Analyzed attack vectors across categories, including AI-based attacks (e.g., deep learning on distorted text) and side-channel attacks (e.g., timing, design flaws).
• Evaluated security, usability, and compatibility trade-offs by comparing schemes across form factors, sensor dependencies, and client-side validation risks.
• Assessed privacy risks associated with collecting behavioral and sensor data, and proposed mitigation strategies such as sending only test results.
• Identified gaps in existing research, particularly the lack of usability studies for behavior- and sensor-based CAPTCHAs, and the need for lightweight, hardware-agnostic solutions.

Experimental results

Research questions

• RQ1 What are the key categories of CAPTCHA schemes developed over the past 20 years, and how do they differ in design and purpose?
• RQ2 How have attack methods evolved to break traditional CAPTCHA schemes, and what are the most effective techniques used against them?
• RQ3 What are the security, usability, and privacy limitations of behavior-based and sensor-based CAPTCHA schemes compared to traditional ones?
• RQ4 Why are current CAPTCHA schemes not uniformly compatible across devices, and what are the challenges in enabling them on desktops and laptops?
• RQ5 What are the open research challenges in designing next-generation CAPTCHAs that are secure, user-friendly, and privacy-preserving?

Key findings

• The majority of traditional CAPTCHA schemes—especially text-based and image-based—have been broken by modern AI, with deep learning models achieving up to 99.8% accuracy on distorted text.
• Side-channel attacks exploiting design weaknesses, such as timing or rendering flaws, have successfully broken several CAPTCHA schemes without solving the underlying AI problem.
• Behavior-based CAPTCHAs, such as Google’s reCAPTCHA v3, are now dominant in production systems, offering frictionless challenges but relying on device-specific interaction patterns.
• Sensor-based CAPTCHAs, which use data from accelerometers or gyroscopes, are highly effective but limited to mobile and wearable devices, raising portability and compatibility concerns.
• Privacy risks are significant in behavior- and sensor-based CAPTCHAs due to the collection of user behavioral and biometric-like data, especially when sent to remote servers without client-side protection.
• No existing study fully evaluates the usability of behavior- or sensor-based CAPTCHAs, highlighting a critical research gap despite their growing deployment.