[Paper Review] How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection

Biyang Guo, Xin Zhang+6 more|arXiv (Cornell University)|Jan 18, 2023

Artificial Intelligence in Healthcare and Education292 citations

TL;DR

This paper introduces HC3, a large Human ChatGPT Comparison Corpus, analyzes linguistic and perceptual gaps between ChatGPT and human experts, and develops detectors to distinguish AI-generated vs human-generated content, with open-source data and models.

ABSTRACT

The introduction of ChatGPT has garnered widespread attention in both academic and industrial communities. ChatGPT is able to respond effectively to a wide range of human questions, providing fluent and comprehensive answers that significantly surpass previous public chatbots in terms of security and usefulness. On one hand, people are curious about how ChatGPT is able to achieve such strength and how far it is from human experts. On the other hand, people are starting to worry about the potential negative impacts that large language models (LLMs) like ChatGPT could have on society, such as fake news, plagiarism, and social security issues. In this work, we collected tens of thousands of comparison responses from both human experts and ChatGPT, with questions ranging from open-domain, financial, medical, legal, and psychological areas. We call the collected dataset the Human ChatGPT Comparison Corpus (HC3). Based on the HC3 dataset, we study the characteristics of ChatGPT's responses, the differences and gaps from human experts, and future directions for LLMs. We conducted comprehensive human evaluations and linguistic analyses of ChatGPT-generated content compared with that of humans, where many interesting results are revealed. After that, we conduct extensive experiments on how to effectively detect whether a certain text is generated by ChatGPT or humans. We build three different detection systems, explore several key factors that influence their effectiveness, and evaluate them in different scenarios. The dataset, code, and models are all publicly available at https://github.com/Hello-SimpleAI/chatgpt-comparison-detection.

Motivation & Objective

• Construct a large-scale HC3 dataset containing questions and answers from human experts and ChatGPT across diverse domains.
• Perform human evaluations and linguistic analyses to characterize differences between human and ChatGPT responses.
• Develop and evaluate detectors that distinguish AI-generated from human-generated text across multiple scenarios.
• Open-source the dataset, evaluations, and detection models to support future research and platform regulation.

Proposed method

• Assemble HC3 from public QA datasets and wiki sources to collect questions and both human and ChatGPT answers.
• Conduct multi-faceted human evaluations including expert/amateur Turing tests and helpfulness assessments.
• Carry out linguistic analyses (POS, dependency, sentiment, perplexity) to compare human vs ChatGPT outputs.
• Train and evaluate three detection systems (GLTR-based logistic regression, RoBERTa single-text, RoBERTa QA) under varied data conditions.
• Examine robustness and generalization by creating six data variants (raw/filtered, full/sent, sentences/mixed) and testing OOD performance.

Experimental results

Research questions

• RQ1 How do ChatGPT and human experts differ in linguistic and stylistic characteristics across domains?
• RQ2 What factors influence the detectability of AI-generated content compared to human-written text?
• RQ3 How robust are different AI-content detectors under various data conditions and our six variant datasets?
• RQ4 What guidance can be provided for future LLM development and platform governance based on HC3 analyses?

Key findings

• ChatGPT answers are generally more organized, longer, and more neutral than human answers, with distinct POS and dependency patterns.
• Humans tend to use a larger vocabulary and more diverse expressions, while ChatGPT exhibits lower perplexity and more consistent stylistic patterns.
• In human evaluations, it is easier to distinguish ChatGPT when a pair of answers is shown than with a single answer, and experts outperform amateurs in detection tasks.
• ChatGPT content is often perceived as more helpful in many domains (finance, psychology) but less so in the medical domain due to overly lengthy or cautious responses.
• Linguistic analyses show ChatGPT uses more nouns, verbs, determiners, and conjunctions, while humans exhibit more varied emotion and subjective expressions.