[Paper Review] A Categorical Archive of ChatGPT Failures

Ali Borji|arXiv (Cornell University)|Feb 6, 2023

Artificial Intelligence in Healthcare and Education91 citations

TL;DR

The paper analyzes ChatGPT’s failures by organizing them into eleven categories, illustrating limitations across reasoning, math, coding, bias, factual accuracy, and more to guide future improvements.

ABSTRACT

Large language models have been demonstrated to be valuable in different fields. ChatGPT, developed by OpenAI, has been trained using massive amounts of data and simulates human conversation by comprehending context and generating appropriate responses. It has garnered significant attention due to its ability to effectively answer a broad range of human inquiries, with fluent and comprehensive answers surpassing prior public chatbots in both security and usefulness. However, a comprehensive analysis of ChatGPT's failures is lacking, which is the focus of this study. Eleven categories of failures, including reasoning, factual errors, math, coding, and bias, are presented and discussed. The risks, limitations, and societal implications of ChatGPT are also highlighted. The goal of this study is to assist researchers and developers in enhancing future language models and chatbots.

Motivation & Objective

• Identify and categorize common failure modes of ChatGPT to create a reference for evaluating progress over time.
• Highlight risks, limitations, and societal implications of large language models to guide responsible development.
• Provide a dataset-like reference of failure examples to aid model training and testing.

Proposed method

• Compile failure examples sourced from public demonstrations and prior work.
• Group failures into eleven categories including reasoning, logic, math, factual errors, bias, humor, and coding.
• Discuss each category with representative instances and associated implications.

Experimental results

Research questions

• RQ1 What categories best capture the typical failure modes of ChatGPT across diverse tasks?
• RQ2 How do these failures manifest in reasoning, arithmetic, factual accuracy, and social biases?
• RQ3 What are the implications of these failures for safety, ethics, and future model development?

Key findings

• Eleven categories of ChatGPT failures are identified and discussed.
• Failures span reasoning, logic, math and arithmetic, factual errors, bias and discrimination, wit and humor, coding, and more.
• The analysis emphasizes limitations like lack of a world model, susceptibility to hallucinations, and the need for standardized benchmarks.
• Examples show improvements over time in some areas (e.g., physical reasoning), but many challenges persist across categories.
• The work proposes the archive as a reference point for evaluating model progress and for generating synthetic data for training and testing.