[Paper Review] ChatGPT (Feb 13 Version) is a Chinese Room

Maurice HT Ling|arXiv (Cornell University)|Feb 19, 2023

Artificial Intelligence in Healthcare and Education13 citations

TL;DR

The paper argues that ChatGPT (Feb 13 version) behaves like a Chinese Room, showing causal reasoning errors, potential hallucinations, and false references that limit its learning utility.

ABSTRACT

ChatGPT has gained both positive and negative publicity after reports suggesting that it is able to pass various professional and licensing examinations. This suggests that ChatGPT may pass Turing Test in the near future. However, a computer program that passing Turing Test can either mean that it is a Chinese Room or artificially conscious. Hence, the question of whether the current state of ChatGPT is more of a Chinese Room or approaching artificial consciousness remains. Here, I demonstrate that the current version of ChatGPT (Feb 13 version) is a Chinese Room. Despite potential evidence of cognitive connections, ChatGPT exhibits critical errors in causal reasoning. At the same time, I demonstrate that ChatGPT can generate all possible categorical responses to the same question and response with erroneous examples; thus, questioning its utility as a learning tool. I also show that ChatGPT is capable of artificial hallucination, which is defined as generating confidently wrong replies. It is likely that errors in causal reasoning leads to hallucinations. More critically, ChatGPT generates false references to mimic real publications. Therefore, its utility is cautioned.

Motivation & Objective

• Motivate whether current ChatGPT resembles a Chinese Room or approaches artificial consciousness.
• Identify and illustrate critical reasoning and factual errors in ChatGPT’s outputs.
• Assess implications of potential hallucinations and false citations for learning and trust in AI systems.

Proposed method

• Present a qualitative critique of ChatGPT (Feb 13 version) across several tasks and prompts.
• Demonstrate causal reasoning errors through examples.
• Show that ChatGPT can generate all possible categorical responses and provide erroneous exemplars.
• Illustrate artificial hallucination defined as confidently wrong replies.
• Show that ChatGPT can generate false references to mimic real publications.

Experimental results

Research questions

• RQ1 Does the Feb 13 version of ChatGPT exhibit characteristics of a Chinese Room rather than genuine understanding?
• RQ2 What evidence of causal reasoning errors does ChatGPT display?
• RQ3 Does ChatGPT exhibit artificial hallucinations and false references, and how do these affect its utility as a learning tool?
• RQ4 Can ChatGPT output erroneous or misleading exemplars while maintaining plausible fluent language?

Key findings

• ChatGPT (Feb 13 version) displays critical errors in causal reasoning.
• ChatGPT can generate all possible categorical responses to the same question with erroneous exemplars.
• ChatGPT is capable of artificial hallucination—confidently wrong replies.
• ChatGPT generates false references that mimic real publications, limiting its utility as a learning tool.