[Paper Review] GPTEval: A Survey on Assessments of ChatGPT and GPT-4

Rui Mao, Guanyi Chen+3 more|arXiv (Cornell University)|Aug 24, 2023

Artificial Intelligence in Healthcare and Education24 citations

TL;DR

A comprehensive survey of how ChatGPT and GPT-4 have been evaluated across language, reasoning, scientific knowledge, and ethics, highlighting strengths, weaknesses, and methodological concerns.

ABSTRACT

The emergence of ChatGPT has generated much speculation in the press about its potential to disrupt social and economic systems. Its astonishing language ability has aroused strong curiosity among scholars about its performance in different domains. There have been many studies evaluating the ability of ChatGPT and GPT-4 in different tasks and disciplines. However, a comprehensive review summarizing the collective assessment findings is lacking. The objective of this survey is to thoroughly analyze prior assessments of ChatGPT and GPT-4, focusing on its language and reasoning abilities, scientific knowledge, and ethical considerations. Furthermore, an examination of the existing evaluation methods is conducted, offering several recommendations for future research in evaluating large language models.

Motivation & Objective

• Assess the language proficiency and reasoning abilities of ChatGPT and GPT-4 across diverse tasks and disciplines.
• Summarize findings on scientific knowledge and domain-specific performance.
• Identify ethical considerations and biases in current evaluations and deployments.
• Critically analyze evaluation methodologies and provide recommendations for future work.

Proposed method

• Review quantitative assessments of ChatGPT and GPT-4 across multiple domains and tasks.
• Analyze results related to language understanding, generation, and reasoning capabilities.
• Critically examine evaluation methods, prompts, and data leakage concerns that affect fairness.
• Synthesize findings on scientific knowledge across formal and natural sciences.
• Discuss ethical considerations including fairness, robustness, reliability, and data privacy.

Experimental results

Research questions

• RQ1 What are the demonstrated language and reasoning strengths and limitations of ChatGPT and GPT-4 across tasks and disciplines?
• RQ2 How do ChatGPT and GPT-4 perform on scientific knowledge domains compared to expert models or humans?
• RQ3 What reliability and fairness issues arise from current evaluation methodologies for large language models?
• RQ4 What ethical considerations emerge from using GPT models in real-world contexts, including data leakage and prompt influence?] ,
• RQ5 key_findings and further_analysis_note

Key findings

• ChatGPT and GPT-4 show strong language understanding and generation but lag behind expert models in domain-specific knowledge.
• GPT-4 and ChatGPT perform well on many science-related questions but can fail on questions requiring multi-step reasoning.
• Evaluation methods are often unreliable due to prompt engineering and dataset choices, with potential data leakage affecting fairness.
• Prompt design and benchmarking choices heavily influence comparative results across models and tasks.
• GPT-4 achieves near-human performance on some exams like computer science and law, while still showing gaps in other areas and safety concerns.