[Paper Review] Large Language Models are as persuasive as humans, but how? About the cognitive effort and moral-emotional language of LLM arguments

Carlos Carrasco-Farré|arXiv (Cornell University)|Apr 14, 2024

European and International Law Studies12 citations

TL;DR

The paper compares LLM-generated and human-generated arguments, showing LLMs impose higher cognitive effort and use more moral language, with no difference in emotional content.

ABSTRACT

Large Language Models (LLMs) are already as persuasive as humans. However, we know very little about how they do it. This paper investigates the persuasion strategies of LLMs, comparing them with human-generated arguments. Using a dataset of 1,251 participants in an experiment, we analyze the persuasion strategies of LLM-generated and human-generated arguments using measures of cognitive effort (lexical and grammatical complexity) and moral-emotional language (sentiment and moral analysis). The study reveals that LLMs produce arguments that require higher cognitive effort, exhibiting more complex grammatical and lexical structures than human counterparts. Additionally, LLMs demonstrate a significant propensity to engage more deeply with moral language, utilizing both positive and negative moral foundations more frequently than humans. In contrast with previous research, no significant difference was found in the emotional content produced by LLMs and humans. These findings contribute to the discourse on AI and persuasion, highlighting the dual potential of LLMs to both enhance and undermine informational integrity through communication strategies for digital persuasion.

Motivation & Objective

• Investigate how LLMs persuade compared with humans.
• Analyze cognitive effort in arguments via lexical and grammatical complexity.
• Examine the use of moral-emotional language in LLM and human arguments.

Proposed method

• Use a dataset of 1,251 participants in an experiment.
• Analyze LLM-generated and human-generated arguments for cognitive effort via lexical/grammatical complexity.
• Assess moral language with moral foundation analysis and sentiment analysis.
• Compare emotional content across LLM and human arguments.

Experimental results

Research questions

• RQ1 Do LLM-generated arguments differ from human arguments in cognitive effort indicators?
• RQ2 Do LLM arguments employ more moral language than human arguments?
• RQ3 Is there a difference in emotional content between LLM and human arguments?

Key findings

• LLMs produce arguments requiring higher cognitive effort through more complex grammar and lexicon.
• LLMs demonstrate a significant propensity to engage more deeply with moral language, using positive and negative moral foundations more frequently than humans.
• There is no significant difference in emotional content between LLM and human arguments.