[Paper Review] Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks

Yiming Zhu, Peixian Zhang+3 more|arXiv (Cornell University)|Apr 20, 2023

Topic ModelingComputer Science65 citations

TL;DR

ChatGPT can reproduce human-generated annotations across five social computing tasks with an average accuracy of 0.609, though performance varies substantially by task and label.

ABSTRACT

The release of ChatGPT has uncovered a range of possibilities whereby large language models (LLMs) can substitute human intelligence. In this paper, we seek to understand whether ChatGPT has the potential to reproduce human-generated label annotations in social computing tasks. Such an achievement could significantly reduce the cost and complexity of social computing research. As such, we use ChatGPT to relabel five seminal datasets covering stance detection (2x), sentiment analysis, hate speech, and bot detection. Our results highlight that ChatGPT does have the potential to handle these data annotation tasks, although a number of challenges remain. ChatGPT obtains an average accuracy 0.609. Performance is highest for the sentiment analysis dataset, with ChatGPT correctly annotating 64.9% of tweets. Yet, we show that performance varies substantially across individual labels. We believe this work can open up new lines of analysis and act as a basis for future research into the exploitation of ChatGPT for human annotation tasks.

Motivation & Objective

• Assess whether ChatGPT can reproduce human-generated annotations in social computing tasks.
• Compare ChatGPT-generated labels to ground-truth human annotations across multiple datasets.
• Analyze per-task and per-label performance to identify strengths and limitations of using ChatGPT for data labeling.
• Provide insights to guide future use of LLMs in human annotation and crowdsourcing contexts.

Proposed method

• Select five English-language, human-annotated Twitter datasets covering stance, hate speech, sentiment, bot detection, and Russo-Ukrainian sentiment.
• Use OpenAI GPT-3.5-turbo to annotate Tweets with a prompt template: given a topic and a set of labels, classify the Tweet and provide an explanation.
• Extract ChatGPT’s primary label from the first sentence of the response and use the full text to derive explanations.
• Evaluate ChatGPT annotations against human ground truth using weighted F1-score, precision, and recall.

Experimental results

Research questions

• RQ1 Can ChatGPT reproduce human-annotated labels on diverse social computing tasks?
• RQ2 How does ChatGPT performance vary across tasks and across individual label categories?
• RQ3 What are the strengths and limitations of using ChatGPT as an annotation tool in terms of precision and recall for each label?

Key findings

• Average ChatGPT annotation accuracy across five tasks is 0.609 (SD 0.032).
• Sentiment analysis yields the highest task accuracy at 0.649 with 64.9% correct labels.
• Hate speech task shows lower overall performance at 0.571, with precision issues notably for the Hate label (0.353).
• Bot detection task achieves 0.639 accuracy but exhibits large label gaps (Human 0.748 F1 vs Bot 0.364 F1).
• Russo-Ukrainian sentiment shows lowest task performance at 0.573 with notable Pro-Russia vs Pro-Ukraine label disparities (0.733 vs 0.429 F1).
• ChatGPT’s label performance varies heavily by individual labels within tasks, indicating limits for uniform annotation quality.