[Paper Review] Human heuristics for AI-generated language are flawed

Maurice Jakesch, Jeffrey T. Hancock+1 more|arXiv (Cornell University)|Jun 15, 2022

Ethics and Social Impacts of AI18 citations

TL;DR

This study demonstrates that humans consistently fail to detect AI-generated self-presentations across professional, hospitality, and dating contexts, despite relying on flawed heuristics like first-person pronouns, contractions, and family references—leading to AI text being perceived as 'more human than human.' The findings reveal that these intuitive cues are predictable and manipulable, undermining human ability to discern synthetic language.

ABSTRACT

Human communication is increasingly intermixed with language generated by AI. Across chat, email, and social media, AI systems suggest words, complete sentences, or produce entire conversations. AI-generated language is often not identified as such but presented as language written by humans, raising concerns about novel forms of deception and manipulation. Here, we study how humans discern whether verbal self-presentations, one of the most personal and consequential forms of language, were generated by AI. In six experiments, participants (N = 4,600) were unable to detect self-presentations generated by state-of-the-art AI language models in professional, hospitality, and dating contexts. A computational analysis of language features shows that human judgments of AI-generated language are hindered by intuitive but flawed heuristics such as associating first-person pronouns, use of contractions, or family topics with human-written language. We experimentally demonstrate that these heuristics make human judgment of AI-generated language predictable and manipulable, allowing AI systems to produce text perceived as "more human than human." We discuss solutions, such as AI accents, to reduce the deceptive potential of language generated by AI, limiting the subversion of human intuition.

Motivation & Objective

• To investigate whether humans can reliably detect AI-generated self-presentations in natural language contexts.
• To identify which intuitive heuristics people use when judging whether language is human- or AI-written.
• To examine how these heuristics lead to systematic errors in human judgment of AI-generated text.
• To evaluate whether AI systems can exploit these heuristics to produce text perceived as more human than actual human writing.
• To propose solutions like AI accents to reduce deception risks in AI-generated language.

Proposed method

• Conducted six controlled experiments with 4,600 participants across diverse contexts: professional, hospitality, and dating self-presentations.
• Used state-of-the-art language models to generate AI-written self-presentations that mirrored human writing in style and content.
• Employed computational analysis of linguistic features to identify patterns associated with perceived humanness.
• Measured human detection accuracy and analyzed response patterns to isolate reliance on specific heuristics (e.g., pronouns, contractions, family topics).
• Used statistical modeling to test whether heuristic-based judgments were predictable and manipulable by AI.
• Proposed and evaluated the concept of 'AI accents'—deliberate stylistic markers—to increase transparency and reduce deception.

Experimental results

Research questions

• RQ1 Can humans reliably detect AI-generated self-presentations in real-world communication contexts?
• RQ2 Which intuitive heuristics do humans use when judging whether language is written by a human or AI?
• RQ3 To what extent are these heuristics flawed and systematically exploitable by AI to produce text perceived as more human than human?
• RQ4 How do linguistic features such as first-person pronouns, contractions, and family references influence human perception of authenticity?
• RQ5 Can 'AI accents' reduce the deceptive potential of AI-generated language by making synthetic text more identifiable?

Key findings

• Participants failed to detect AI-generated self-presentations at rates significantly above chance, with detection accuracy averaging below 50% across all contexts.
• Humans consistently associated first-person pronouns (e.g., 'I', 'me'), contractions (e.g., 'don’t', 'I’m'), and references to family with human-written language, despite these features being common in AI output.
• These heuristics were predictable and exploitable: AI systems could generate text that scored higher on 'humanness' metrics than actual human writing.
• The study demonstrated that AI-generated text could be perceived as 'more human than human' by exploiting human cognitive biases.
• Computational analysis confirmed that the most salient linguistic features linked to perceived humanness were also the most manipulable by AI.
• The findings support the need for transparency mechanisms such as 'AI accents' to counter deceptive language generation.