[Paper Review] Machine Psychology

Thilo Hagendorff, Dasgupta, Ishita+6 more|arXiv (Cornell University)|Mar 24, 2023

Topic Modeling67 citations

TL;DR

The paper argues for studying large language models through behavioral experiments inspired by psychology to gain computational insights into emergent abilities and behavioral patterns, beyond standard performance benchmarks.

ABSTRACT

Large language models (LLMs) show increasingly advanced emergent capabilities and are being incorporated across various societal domains. Understanding their behavior and reasoning abilities therefore holds significant importance. We argue that a fruitful direction for research is engaging LLMs in behavioral experiments inspired by psychology that have traditionally been aimed at understanding human cognition and behavior. In this article, we highlight and summarize theoretical perspectives, experimental paradigms, and computational analysis techniques that this approach brings to the table. It paves the way for a "machine psychology" for generative artificial intelligence (AI) that goes beyond performance benchmarks and focuses instead on computational insights that move us toward a better understanding and discovery of emergent abilities and behavioral patterns in LLMs. We review existing work taking this approach, synthesize best practices, and highlight promising future directions. We also highlight the important caveats of applying methodologies designed for understanding humans to machines. We posit that leveraging tools from experimental psychology to study AI will become increasingly valuable as models evolve to be more powerful, opaque, multi-modal, and integrated into complex real-world settings.

Motivation & Objective

• Motivate applying experimental psychology methods to LLMs to understand cognition and behavior.
• Synthesize theoretical perspectives, experimental paradigms, and computational analysis techniques for machine psychology.
• Highlight caveats when transferring human-centric methodologies to machines.
• Chart future directions for robust, interpretable insights into emergent AI capabilities.

Proposed method

• Review existing work that uses psychology-inspired experimentation with LLMs.
• Summarize theoretical perspectives, experimental paradigms, and computational analysis techniques.
• Synthesize best practices and identify caveats for cross-domain methodology transfer.
• Propose future research directions for machine psychology as models evolve.

Experimental results

Research questions

• RQ1 What experimental paradigms from psychology can be effectively adapted to study LLMs?
• RQ2 What computational analysis techniques best reveal emergent abilities in generative AI beyond performance benchmarks?
• RQ3 What caveats arise when applying human-centric methods to machines, and how can they be mitigated?
• RQ4 What future directions will enhance understanding of AI behavior in complex real-world settings?

Key findings

• LLMs exhibit emergent behaviors that can be probed with psychology-inspired experiments.
• Integrating experimental psychology tools can yield computational insights beyond standard benchmarks.
• There are important caveats when applying human-oriented methodologies to machines that require careful adaptation.
• The approach is valuable as models become more powerful, opaque, multi-modal, and integrated into real-world contexts.