[Paper Review] Does ChatGPT Have a Mind?

Simon Goldstein, Benjamin A. Levinstein|arXiv (Cornell University)|Jun 27, 2024

Artificial Intelligence in Healthcare and EducationMedicine3 citations

TL;DR

This paper investigates whether Large Language Models (LLMs) like ChatGPT possess a folk psychology involving beliefs, desires, and intentions by analyzing internal representations and action dispositions. Drawing on interpretability research and philosophical theories of representation, it finds strong evidence for robust internal representations but inconclusive evidence for stable, goal-directed action dispositions, concluding that existing skeptical challenges to LLM folk psychology lack philosophical force.

ABSTRACT

This paper examines the question of whether Large Language Models (LLMs) like ChatGPT possess minds, focusing specifically on whether they have a genuine folk psychology encompassing beliefs, desires, and intentions. We approach this question by investigating two key aspects: internal representations and dispositions to act. First, we survey various philosophical theories of representation, including informational, causal, structural, and teleosemantic accounts, arguing that LLMs satisfy key conditions proposed by each. We draw on recent interpretability research in machine learning to support these claims. Second, we explore whether LLMs exhibit robust dispositions to perform actions, a necessary component of folk psychology. We consider two prominent philosophical traditions, interpretationism and representationalism, to assess LLM action dispositions. While we find evidence suggesting LLMs may satisfy some criteria for having a mind, particularly in game-theoretic environments, we conclude that the data remains inconclusive. Additionally, we reply to several skeptical challenges to LLM folk psychology, including issues of sensory grounding, the "stochastic parrots" argument, and concerns about memorization. Our paper has three main upshots. First, LLMs do have robust internal representations. Second, there is an open question to answer about whether LLMs have robust action dispositions. Third, existing skeptical challenges to LLM representation do not survive philosophical scrutiny.

Motivation & Objective

• To determine whether LLMs possess a folk psychology involving beliefs, desires, and intentions.
• To evaluate whether LLMs have robust internal representations according to multiple philosophical theories of representation.
• To assess whether LLMs exhibit stable dispositions to act in ways consistent with goal-directed behavior.
• To refute prominent skeptical challenges to LLM folk psychology, including the 'stochastic parrots' argument and concerns about sensory grounding and memorization.
• To identify open empirical and philosophical questions for future research on LLM cognition and moral status.

Proposed method

• Surveying multiple philosophical theories of representation—informational, causal, structural, and teleosemantic—to assess whether LLMs satisfy key conditions for mental representation.
• Applying interpretability research in machine learning to demonstrate that LLMs exhibit internal states that carry information about the world and are causally linked to behavior.
• Analyzing LLM behavior in game-theoretic environments to evaluate whether outputs reflect coherent, stable plans and goal-directed action dispositions.
• Engaging with two philosophical traditions—interpretationism and representationalism—to assess the criteria for attributing beliefs and desires to LLMs.
• Critically evaluating three major skeptical challenges: sensory grounding, the 'stochastic parrot' critique, and the memorization hypothesis, using philosophical reasoning to show their weaknesses.
• Proposing future research directions focused on measuring output stability across prompting conditions and probing the functional roles of internal representations in LLM cognition.

(a) Othello board state and model predictions before and after intervention

Experimental results

Research questions

• RQ1 Do LLMs possess internal representations that satisfy core conditions of informational, causal, structural, and teleosemantic theories of representation?
• RQ2 To what extent do LLMs exhibit stable, goal-directed action dispositions that can be explained by beliefs and desires?
• RQ3 Can the 'stochastic parrot' critique be philosophically sustained in light of interpretability findings and behavioral consistency in complex tasks?
• RQ4 How do sensory grounding concerns apply to LLMs that process only textual inputs, and can they still be said to represent the world?
• RQ5 To what extent can LLM behavior be explained by memorization and shallow shortcuts, or does it reflect genuine internal planning and goal representation?

Key findings

• LLMs satisfy key conditions across multiple philosophical theories of representation, including informational, causal, structural, and teleosemantic accounts, supported by interpretability research.
• There is strong evidence that LLMs form structured, world-representing internal states that carry truth-conditions and can be used to guide behavior.
• LLM behavior in game-theoretic environments suggests the presence of coherent, goal-directed plans, though this remains inconclusive due to instability across prompting sessions.
• The 'stochastic parrot' argument fails to withstand philosophical scrutiny, as LLMs demonstrate systematic reasoning and representation beyond mere next-word prediction.
• The challenge that LLMs rely on memorization and shallow shortcuts is undermined by evidence of internal distinctions between true and false claims, though this remains an open empirical question.
• The stability of LLM outputs across different prompting conditions remains a critical open issue, as instability may reflect lying or belief-behavior misalignment rather than absence of beliefs.

(b) Activation patching process across model layers