[Paper Review] Cognitive Model Priors for Predicting Human Decisions

David Bourgin, Joshua C. Peterson+3 more|arXiv (Cornell University)|May 22, 2019

Complex Systems and Decision Making34 citations

TL;DR

The paper introduces cognitive model priors by pretraining neural networks on synthetic data from cognitive models (BEAST variants) and fine-tuning on real data, achieving state-of-the-art prediction of human choices under uncertainty and introducing a large choices13k dataset.

ABSTRACT

Human decision-making underlies all economic behavior. For the past four decades, human decision-making under uncertainty has continued to be explained by theoretical models based on prospect theory, a framework that was awarded the Nobel Prize in Economic Sciences. However, theoretical models of this kind have developed slowly, and robust, high-precision predictive models of human decisions remain a challenge. While machine learning is a natural candidate for solving these problems, it is currently unclear to what extent it can improve predictions obtained by current theories. We argue that this is mainly due to data scarcity, since noisy human behavior requires massive sample sizes to be accurately captured by off-the-shelf machine learning methods. To solve this problem, what is needed are machine learning models with appropriate inductive biases for capturing human behavior, and larger datasets. We offer two contributions towards this end: first, we construct "cognitive model priors" by pretraining neural networks with synthetic data generated by cognitive models (i.e., theoretical models developed by cognitive psychologists). We find that fine-tuning these networks on small datasets of real human decisions results in unprecedented state-of-the-art improvements on two benchmark datasets. Second, we present the first large-scale dataset for human decision-making, containing over 240,000 human judgments across over 13,000 decision problems. This dataset reveals the circumstances where cognitive model priors are useful, and provides a new standard for benchmarking prediction of human decisions under uncertainty.

Motivation & Objective

• Motivate improving prediction of human decision-making under uncertainty beyond traditional theories.
• Develop cognitive model priors by translating cognitive theories into synthetic training data for neural networks.
• Demonstrate improved prediction on CPC15/CPC18 benchmarks using pretrained networks fine-tuned on real data.
• Create large-scale public datasets (synth15, synth18, choices13k) to benchmark human decision prediction.

Proposed method

• Generate synthetic problem sets from cognitive models BEAST15 and BEAST18 to train neural networks as priors.
• Train neural networks to approximate BEAST predictions on synthetic data and then fine-tune on real human decision data.
• Use Sparse MLP with SET regularization and fine-tuning with a very small learning rate to incorporate priors.
• Evaluate predictions on CPC15 and CPC18 benchmarks against traditional ML models and theory-based baselines.
• Introduce choices13k, a large-scale human risky-choice dataset collected via Mechanical Turk.

Experimental results

Research questions

• RQ1 Can cognitive models provide effective priors to bootstrap machine learning when real data are scarce?
• RQ2 Do pretraining on synthetic cognitive-model data improve predictive accuracy and training efficiency on human decision data?
• RQ3 How do cognitive-model priors perform on established CPC benchmarks compared to theory-only and pure ML approaches?
• RQ4 What is the value and generalization of a large new human decision dataset (choices13k) for benchmarking predictive models?
• RQ5 Are there practical benefits (e.g., faster convergence, data-efficiency) from cognitive priors in training?

Key findings

• Fine-tuning neural networks pretrained on synth15 synth18 cognitive priors yields substantial MSE reductions on CPC15 and CPC18 benchmarks compared to baselines.
• CPC15: cognitive-prior enhanced model achieves MSE 0.53 (×100) versus 0.88 for CPC15 winner and 0.99 for BEAST15.
• CPC18: cognitive-prior enhanced model achieves MSE 0.48 (×100) versus 0.57 for CPC18 winner and 0.70 for BEAST18.
• On CPC18, the synth18-trained MLP with fine-tuning attains 0.48, improving over the CPC18 winner by 0.09 (×100 units).
• The choices13k dataset (242,879 judgments over 13,006 problems) demonstrates that cognitive priors reduce prediction error and speed up training, especially with limited human data.