[Paper Review] Excitements and Concerns in the Post-ChatGPT Era: Deciphering Public Perception of AI through Social Media Analysis

Weihong Qi, Jinsheng Pan+2 more|arXiv (Cornell University)|Jul 11, 2023

Misinformation and Its Impacts10 citations

TL;DR

The paper analyzes public perception of AI on Reddit post-ChatGPT using BERTopic for topic modeling, zero-shot sentiment analysis, and LIWC-based linguistic features across 388 subreddits and 33,912 comments.

ABSTRACT

As AI systems become increasingly prevalent in various aspects of daily life, gaining a comprehensive understanding of public perception towards these AI systems has become increasingly essential for several reasons such as ethical considerations, user experience, fear, disinformation, regulation, collaboration, and co-creation. In this study, we investigate how mass social media users perceive the recent rise of AI frameworks such as ChatGPT. We collect a total of 33,912 comments in 388 unique subreddits spanning from November 30, 2022 to June 8, 2023 using a list of AI-related keywords. We employ BERTopic to uncover the major themes regarding AI on Reddit. Additionally, we seek to gain deeper insights into public opinion by examining the distribution of topics across different subreddits. We observe that technology-related subreddits predominantly focus on the technical aspects of AI models. On the other hand, non-tech subreddits show greater interest in social issues such as concerns about job replacement or furlough. We leverage zero-shot prompting to analyze the sentiment and perception of AI among individual users. Through a comprehensive sentiment and emotion analysis, we discover that tech-centric communities exhibit greater polarization compared to non-tech communities when discussing AI topics. This research contributes to our broader understanding of public opinion surrounding artificial intelligence.

Motivation & Objective

• Understand how mass social media users perceive the rise of AI frameworks like ChatGPT.
• Identify major themes and topic distributions across diverse Reddit communities.
• Examine sentiment and emotional tone across topics and subreddits to reveal differences between tech-centric and non-tech groups.

Proposed method

• Collect 33,912 Reddit comments from 388 subreddits (Nov 30, 2022 – Jun 8, 2023) using AI-related keywords.
• Apply BERTopic with all-MiniLM-L6-v2 embeddings to extract 232 topics (16 outliers removed).
• Use zero-shot prompting (gpt-3.5-turbo) for sentiment labels (positive, neutral, negative) and a keyword-based aggregation to assign final sentiment.
• Perform LIWC-based linguistic analysis to obtain Tone, Emotion, Prosocial, and Conflict metrics.
• Run linear regression to compare LIWC attributes between tech-centric and non-tech subreddits (Tech as a binary variable).
• Conduct cross-subreddit analysis to map topic distributions and sentiment across communities.

Experimental results

Research questions

• RQ1 RQ1: What topics characterize AI discussions on Reddit and how do they vary across subreddits?
• RQ2 RQ2: What is the prevailing sentiment for the most discussed topics, and do sentiments differ among subreddits?

Key findings

• Top topics include Consciousness and Intelligence of AI, AI in Gaming and Strategy, AI Model and Prompt Engineering, OpenAI and AI Industry, and AI and Job Automation.
• Tech-centric subreddits focus more on technical aspects, while non-tech subreddits emphasize social implications like job replacement and privacy.
• Tech communities exhibit higher positive tone and higher overall emotional expression, indicating greater polarization in sentiment.
• Tech subreddits show higher Prosocial scores, suggesting more signaling of helping/caring in AI discussions.
• Across subreddits, several topics show higher positive sentiment (e.g., AI in Gaming and Strategy, The Potential Impact of AI on Society) while others show more negative sentiment (e.g., Consciousness and Intelligence of AI, AI Industry, Job Automation).
• Cross-subreddit analysis reveals three topics (Consciousness/Intelligence, OpenAI/AI Industry, AI and Job Automation) as particularly prevalent across multiple subreddits.