[Paper Review] Computing a human-like reaction time metric from stable recurrent vision models

Lore Goetschalckx, Lakshmi Narasimhan Govindarajan+4 more|arXiv (Cornell University)|Jun 20, 2023

Visual perception and processing mechanismsNeuroscience65 references8 citations

TL;DR

The paper introduces a stimulus-computable reaction-time proxy, xi_cRNN, derived from stable recurrent vision models trained with evidential deep learning, and shows its alignment with human RT patterns across four visual tasks.

ABSTRACT

The meteoric rise in the adoption of deep neural networks as computational models of vision has inspired efforts to "align" these models with humans. One dimension of interest for alignment includes behavioral choices, but moving beyond characterizing choice patterns to capturing temporal aspects of visual decision-making has been challenging. Here, we sketch a general-purpose methodology to construct computational accounts of reaction times from a stimulus-computable, task-optimized model. Specifically, we introduce a novel metric leveraging insights from subjective logic theory summarizing evidence accumulation in recurrent vision models. We demonstrate that our metric aligns with patterns of human reaction times for stimulus manipulations across four disparate visual decision-making tasks spanning perceptual grouping, mental simulation, and scene categorization. This work paves the way for exploring the temporal alignment of model and human visual strategies in the context of various other cognitive tasks toward generating testable hypotheses for neuroscience. Links to the code and data can be found on the project page: https://serre-lab.github.io/rnn_rts_site.

Motivation & Objective

• Motivate temporal alignment between neural network dynamics and human visual decision-making.
• Develop a model-driven, stimulus-computable reaction-time metric from cRNNs.
• Demonstrate that the reaction-time metric aligns qualitatively with human RT patterns across multiple tasks.
• Provide a framework to study temporal dynamics and generate neuroscience hypotheses.

Proposed method

• Train stable recurrent vision models (cRNNs) with C-RBP and Evidential Deep Learning (EDL) to obtain a Dirichlet-distributed belief over classes.
• Define the RT metric xi_cRNN as the area under the model uncertainty curve over time, xi_cRNN = integral_0^T Epsilon(t) dt.
• Use attractor dynamics and EDL to obtain time-evolving uncertainty without extra supervision.
• Apply the framework to four tasks to assess alignment with human RT: incremental grouping, visual simulation (Planko), maze path reasoning, and scene categorization.
• Visualize internal dynamics via latent activity h_t and spatial uncertainty maps to interpret decision strategies.

Figure 1 : Computing a reaction time metric from a recurrent vision model. a. A schematic representation of training a cRNN with evidential deep learning (EDL; [ 30 ] ). Model outputs are interpreted as parameters ( $\boldsymbol{\alpha}$ ) of a Dirichlet distribution over class probability estimates

Experimental results

Research questions

• RQ1 Can xi_cRNN capture stimulus-dependent reaction-time patterns observed in humans?
• RQ2 Do stable cRNNs trained with EDL exhibit temporal dynamics that align with human decision times across diverse visual tasks?
• RQ3 Is xi_cRNN predictive of human RT variations due to task difficulty, stimulus structure, or spatial properties?

Key findings

• xi_cRNN tracks stimulus-dependent RT patterns and aligns qualitatively with human RT across four tasks.
• xi_cRNN reveals serial attention-like strategies and spatial anisotropy similar to human data in incremental grouping.
• xi_cRNN correlates with human RT in the Planko task and in maze/path-length conditions, reflecting longer processing for harder stimuli.
• xi_cRNN predicts discriminability-related RT trends in scene categorization and correlates with human RT (r = 0.19, p < .001).
• Stable training with C-RBP yields robust, time-adaptive processing that improves generalization over BPTT-based approaches.
• The framework provides a general-purpose method to compare model dynamics with human temporal processing and to generate neuroscience hypotheses.

Figure 2 : Human versus cRNN temporal alignment on an incremental grouping task. a. Description of the task (inspired by cognitive neuroscience studies [ 47 ] ). b. Visualization of the cRNN dynamics. The two lines represent the average latent trajectories across $1K$ validation stimuli labeled ‘‘ye