[Paper Review] A neural network walks into a lab: towards using deep nets as models for human behavior

Wei Ji, Benjamin Peters|arXiv (Cornell University)|May 2, 2020

Neural dynamics and brain function82 references46 citations

TL;DR

The paper argues for using deep neural networks (DNNs) as models of human behavior in cognitive science, outlines how to train and test them, and discusses evaluation and integration with traditional cognitive models.

ABSTRACT

What might sound like the beginning of a joke has become an attractive prospect for many cognitive scientists: the use of deep neural network models (DNNs) as models of human behavior in perceptual and cognitive tasks. Although DNNs have taken over machine learning, attempts to use them as models of human behavior are still in the early stages. Can they become a versatile model class in the cognitive scientist's toolbox? We first argue why DNNs have the potential to be interesting models of human behavior. We then discuss how that potential can be more fully realized. On the one hand, we argue that the cycle of training, testing, and revising DNNs needs to be revisited through the lens of the cognitive scientist's goals. Specifically, we argue that methods for assessing the goodness of fit between DNN models and human behavior have to date been impoverished. On the other hand, cognitive science might have to start using more complex tasks (including richer stimulus spaces), but doing so might be beneficial for DNN-independent reasons as well. Finally, we highlight avenues where traditional cognitive process models and DNNs may show productive synergy.

Motivation & Objective

• Advocate DNNs as viable candidate models of human behavior alongside traditional cognitive process models.
• Propose a lab-style pipeline where DNNs undergo ecological pre-training, diagnostic-task training, and limited data-driven fitting to human behavior.
• Call for richer task design and stimulus spaces to better capture human-like priors and behavior in DNNs.
• Emphasize evaluation methods that go beyond accuracy, using rich behavioral measures and model-to-human comparisons.

Proposed method

• Argue for a two-stage training paradigm: ecological pre-training followed by training on diagnostic lab tasks, with possible limited fitting to behavioral data.
• Discuss architectural directions toward biological plausibility (e.g., recurrent, feedback, attention, spiking variants) to yield human-like representations.
• Emphasize the creation of ecologically valid and richly varied training sets, including data augmentation and video-like variability to mirror human experience.
• Recommend multi-task training and curriculum learning to foster modularity and generalization akin to human cognition.
• Advocate inter-individual variability by exploring multiple network instances and training regimes to capture human differences.

Experimental results

Research questions

• RQ1 Can DNNs serve as versatile, human-like models across a range of cognitive tasks?
• RQ2 What training and testing pipelines best align DNN behavior with human behavior on diagnostic tasks?
• RQ3 How should goodness-of-fit between DNNs and humans be evaluated beyond simple accuracy?
• RQ4 What role do ecologically valid pre-training, task richness, and multi-task training play in achieving human-like DNN behavior?
• RQ5 How can cognitive process models and DNNs be integrated to leverage strengths of both approaches?

Key findings

• DNNs should be taken seriously as candidate models of behavior, not just as predictive tools.
• A dignified modeling pipeline includes ecological pre-training, diagnostic-task training, and possibly limited fitting to human data.
• Goodness-of-fit requires rich, multi-dimensional behavioral characterization beyond accuracy, including response distributions and timing.
• Task design and training data richness are crucial; ecologically valid, varied tasks help align DNN priors with human cognition.
• Hybrid approaches combining cognitive process models with DNNs offer productive avenues for advancing understanding of cognition.