[论文解读] ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning

Petter Törnberg|arXiv (Cornell University)|Apr 13, 2023

Hate Speech and Cyberbullying Detection被引用 158

一句话总结

对 Twitter 用户的政治派别归属进行分类的评估中，ChatGPT-4 在准确性和可靠性方面优于专家编码员和 MTurk 众包工作者，偏见方面与之相当或更低，采用零样本学习。

ABSTRACT

This paper assesses the accuracy, reliability and bias of the Large Language Model (LLM) ChatGPT-4 on the text analysis task of classifying the political affiliation of a Twitter poster based on the content of a tweet. The LLM is compared to manual annotation by both expert classifiers and crowd workers, generally considered the gold standard for such tasks. We use Twitter messages from United States politicians during the 2020 election, providing a ground truth against which to measure accuracy. The paper finds that ChatGPT-4 has achieves higher accuracy, higher reliability, and equal or lower bias than the human classifiers. The LLM is able to correctly annotate messages that require reasoning on the basis of contextual knowledge, and inferences around the author's intentions - traditionally seen as uniquely human abilities. These findings suggest that LLM will have substantial impact on the use of textual data in the social sciences, by enabling interpretive research at a scale.

研究动机与目标

• 评估 ChatGPT-4 从 Twitter 内容标注政治派别的准确性。
• 将 ChatGPT-4 的表现与专家编码员和 MTurk 众包工作者在使用真实政治人物推文的基线数据上的表现进行比较。
• 评估 ChatGPT-4 相对于人工标注者的可靠性（编码者间一致性）和偏见。
• 使用 2020 年选举期的美国参议员推文的真实数据集作为评估基线。

提出的方法

• 在 2020 年 US 大选前，从美国参议员的推文中筛选出 500 条推文（其中 250 条共和党，250 条民主党），过滤后(不可转发/回复/链接)，长度≥100。
• 通过 API 使用零-shot/少量-shot 提示对每条推文进行分类，并在不同 temperature 下多次运行（低温 0.2 5 次，高温 1.0 5 次，总共 5000 次分类）。
• 将 ChatGPT-4 的结果与 MTurk 众包工作者（Master Qualified US workers，每条推文 10 位标注员，含控制问题）以及两名专家分类器进行比较。
• 计算与真实标签的准确性、Krippendorff’s Alpha 作为编码者间可靠性指标，以及在所有标注者中的偏见（倾向于民主党对共和党的偏向）。

实验结果

研究问题

• RQ1 Can ChatGPT-4 correctly infer political affiliation from tweet content using zero-shot learning?
• RQ2 How does ChatGPT-4's accuracy compare to expert coders and MTurk crowd workers on this task?
• RQ3 What is the reliability (intercoder agreement) of ChatGPT-4 relative to humans?
• RQ4 Is there a bias toward predicting Democrat or Republican, and how does it compare across groups?

主要发现

• ChatGPT-4 achieves higher accuracy than both expert classifiers and MTurk workers.
• ChatGPT-4 shows higher intercoder reliability (Krippendorff’s Alpha) than human coders, especially at lower temperatures.
• All respondent groups (including ChatGPT-4 and experts) show a bias toward predicting Democrat, while MTurk workers exhibit a significantly stronger bias.
• ChatGPT-4's performance is demonstrated across tweets requiring contextual knowledge and inference of author intent.