[Paper Review] Centaur: a foundation model of human cognition

Marcel Binz, Elif Akata+38 more|arXiv (Cornell University)|Oct 26, 2024

Cognitive Science and Education Research5 citations

TL;DR

Centaur is a foundation model of human cognition obtained by finetuning a large language model on Psych-101, enabling it to predict and simulate human behavior across a wide range of experiments and even align with neural data.

ABSTRACT

Establishing a unified theory of cognition has been a major goal of psychology. While there have been previous attempts to instantiate such theories by building computational models, we currently do not have one model that captures the human mind in its entirety. A first step in this direction is to create a model that can predict human behavior in a wide range of settings. Here we introduce Centaur, a computational model that can predict and simulate human behavior in any experiment expressible in natural language. We derived Centaur by finetuning a state-of-the-art language model on a novel, large-scale data set called Psych-101. Psych-101 reaches an unprecedented scale, covering trial-by-trial data from over 60,000 participants performing over 10,000,000 choices in 160 experiments. Centaur not only captures the behavior of held-out participants better than existing cognitive models, but also generalizes to new cover stories, structural task modifications, and entirely new domains. Furthermore, we find that the model's internal representations become more aligned with human neural activity after finetuning. Taken together, our results demonstrate that it is possible to discover computational models that capture human behavior across a wide range of domains. We believe that such models provide tremendous potential for guiding the development of cognitive theories and present a case study to demonstrate this.

Motivation & Objective

• Motivate the pursuit of a unified, domain-general model of human cognition.
• Introduce Psych-101 as a large-scale, trial-by-trial behavioral dataset.
• Demonstrate that Centaur predicts human behavior better than domain-specific models across many experiments.
• Show that Centaur generalizes to new cover stories, task structures, and domains.
• Investigate whether Centaur's internal representations align with human neural activity.

Proposed method

• Finetune a state-of-the-art language model (Llama 3.1 70B) on Psych-101 using quantized low-rank adaptation (QLoRA) with adapters in non-embedding layers.
• Prepare Psych-101 by transcribing 160 psychological experiments into natural language prompts covering trial-by-trial histories.
• Train for one epoch with cross-entropy loss, masking non-human-response tokens, on an A100 GPU for ~5 days.
• Evaluate using a pseudo-R^2 measure to compare Centaur against Llama and domain-specific cognitive models across held-out participants and experiments.
• Conduct open-loop simulations to assess whether Centaur generates human-like trajectories.
• Assess generalization via out-of-distribution tests with modified stories, task structures, and novel domains.
• Analyze neural alignment by predicting fMRI signals from Centaur’s internal representations and comparing with human data.

Figure 2: Performance on Psych-101. a , Pseudo-R 2 values for different models across experiments. A value of zero corresponds to prediction at chance level while a value of one corresponds to perfect predictability of human responses. Missing bars indicate performance below chance level. Centaur ou

Experimental results

Research questions

• RQ1 Can Centaur predict held-out human behavior better than domain-specific cognitive models across a wide range of experiments?
• RQ2 Does Centaur generalize to unseen experiments, cover stories, and task structures, including entirely novel domains?
• RQ3 Do Centaur’s internal representations become more aligned with human neural activity after finetuning?
• RQ4 Is open-loop simulation of Centaur’s behavior consistent with human distributions of trajectories?
• RQ5 How does Centaur fare on out-of-distribution evaluations compared to baseline models?

Key findings

• Centaur outperforms both the base model (Llama) and a collection of domain-specific cognitive models in most experiments.
• Finetuning yields average pseudo-R^2 improvement of about 0.14 over Llama and about 0.18 over domain-specific models on average.
• Centaur’s open-loop simulations produce human-like trajectory distributions, including model-based, model-free, and mixed reinforcement learning patterns.
• Centaur generalizes to modified cover stories, additional task structures, and entirely new domains with robust performance.
• Internal representations of Centaur align more with human neural activity, improving decodability in fMRI analyses across tasks.

Figure 3: Evaluation in different held-out settings. a , Pseudo-R 2 values for the two-step task with a modified cover story [ 24 ] . b , Pseudo-R 2 values for a three-armed bandit experiment [ 25 ] . c , Pseudo-R 2 values for an experiment probing logical reasoning [ 26 ] . Centaur outperforms both