[Paper Review] AI Chat Assistants can Improve Conversations about Divisive Topics

Lisa P. Argyle, Ethan C. Busby+5 more|arXiv (Cornell University)|Feb 14, 2023

Social Media and Politics12 citations

TL;DR

A GPT-3 based chat assistant provides real-time rephrasing suggestions to online conversations about gun policy, increasing perceived understanding and democratic reciprocity, especially for the conversation partner, without changing policy attitudes.

ABSTRACT

A rapidly increasing amount of human conversation occurs online. But divisiveness and conflict can fester in text-based interactions on social media platforms, in messaging apps, and on other digital forums. Such toxicity increases polarization and, importantly, corrodes the capacity of diverse societies to develop efficient solutions to complex social problems that impact everyone. Scholars and civil society groups promote interventions that can make interpersonal conversations less divisive or more productive in offline settings, but scaling these efforts to the amount of discourse that occurs online is extremely challenging. We present results of a large-scale experiment that demonstrates how online conversations about divisive topics can be improved with artificial intelligence tools. Specifically, we employ a large language model to make real-time, evidence-based recommendations intended to improve participants' perception of feeling understood in conversations. We find that these interventions improve the reported quality of the conversation, reduce political divisiveness, and improve the tone, without systematically changing the content of the conversation or moving people's policy attitudes. These findings have important implications for future research on social media, political deliberation, and the growing community of scholars interested in the place of artificial intelligence within computational social science.

Motivation & Objective

• Demonstrate whether AI-based real-time rephrasing can improve perceived understanding in online political conversations.
• Test if such interventions promote democratic reciprocity without altering participants' policy attitudes.
• Assess whether AI guidance affects conversation tone without shifting the topical content.
• Evaluate differential effects when the AI intervention is applied to one participant versus their partner.
• Explore scalability of AI-assisted conversation interventions for divise topics.

Proposed method

• Conduct a large online field experiment with gun-policy debates in the United States.
• Use a GPT-3 powered chat assistant to propose three context-aware rephrasings (restate, validate, politeness) during live conversations.
• Randomly assign one participant per chat to receive AI suggestions while the partner may or may not be treated.
• Measure conversational quality and democratic reciprocity via post-conversation surveys.
• Analyze text to verify tone changes and topic preservation, and use placebo-controlled subgroup analyses for treatment effects.

Figure 1: Treated Conversation Flow : Respondents write messages unimpeded until one partner receives a rephrasing prompt for the first message longer than four words, and every other conversational turn thereafter. The chat assistant intercepts the treated user’s message, using GPT-3 to propose evi

Experimental results

Research questions

• RQ1 Does AI-assisted rephrasing increase participants' perception of being understood in conversations with political opponents?
• RQ2 Do AI-generated interventions enhance democratic reciprocity toward opponents across the political system?
• RQ3 Do these interventions alter the substantive policy attitudes of participants or their partners?
• RQ4 Are effects larger for partners of treated individuals compared to the treated individuals themselves?

Key findings

• AI rephrasings chosen by participants increased politeness, validation, and restatement features in messages.
• Partners of treated individuals reported higher conversational quality and democratic reciprocity.
• Full treatment yielded about a 6 percentage point increase in democratic reciprocity for partners.
• There was no evidence that AI intervention changed participants' gun-policy attitudes.
• Effects were strongest for conversations with higher initial disagreement and persisted only in the short term.

Figure 2: Text analysis of rephrased messages tone : Marginal difference, with 95% confidence intervals, between rephrased message scores on five politeness package features, and baseline scores from participants’ original messages they would have sent had they not chosen the rephrasing.