[Paper Review] Deep learning approaches for neural decoding: from CNNs to LSTMs and spikes to fMRI

Jesse A. Livezey, Joshua I. Glaser|arXiv (Cornell University)|May 19, 2020

EEG and Brain-Computer Interfaces5 citations

TL;DR

This paper reviews deep learning approaches for neural decoding, demonstrating how CNNs, LSTMs, and other architectures improve accuracy in predicting behavior, perception, and cognitive states from neural data—from spikes to fMRI. It highlights the use of pretrained networks as priors for complex outputs like speech and images, and discusses transfer learning and interpretability for scientific insight in neuroscience applications.

ABSTRACT

Decoding behavior, perception, or cognitive state directly from neural signals has applications in brain-computer interface research as well as implications for systems neuroscience. In the last decade, deep learning has become the state-of-the-art method in many machine learning tasks ranging from speech recognition to image segmentation. The success of deep networks in other domains has led to a new wave of applications in neuroscience. In this article, we review deep learning approaches to neural decoding. We describe the architectures used for extracting useful features from neural recording modalities ranging from spikes to EEG. Furthermore, we explore how deep learning has been leveraged to predict common outputs including movement, speech, and vision, with a focus on how pretrained deep networks can be incorporated as priors for complex decoding targets like acoustic speech or images. Deep learning has been shown to be a useful tool for improving the accuracy and flexibility of neural decoding across a wide range of tasks, and we point out areas for future scientific development.

Motivation & Objective

• To review the application of deep learning in neural decoding across diverse neural recording modalities, including spikes, EEG, fMRI, and fNIRS.
• To evaluate how different deep learning architectures (CNNs, RNNs, LSTMs) are suited to specific neural data types and decoding tasks.
• To explore the use of pretrained deep networks as priors for decoding complex outputs such as speech and visual images.
• To examine transfer learning and interpretability techniques for improving generalization and scientific insight in neural decoding.
• To identify open challenges in data efficiency, model interpretability, and integration of deep learning with neuroscience research.

Proposed method

• Utilizes convolutional neural networks (CNNs) to extract hierarchical, translation-invariant features from 1D (spikes, EEG), 2D (fMRI slices), and 3D (volumetric fMRI) neural data.
• Employs recurrent neural networks (RNNs) and LSTMs to model temporal dependencies in neural sequences, enabling flexible processing of variable-length trials.
• Applies transfer learning by fine-tuning models pre-trained on large-scale datasets (e.g., ImageNet) to decode neural activity into complex outputs like images or speech.
• Incorporates weight-sharing across neurons or channels to learn shared, data-driven features, reducing overfitting on limited neurodata.
• Uses attention mechanisms and feature attribution methods to interpret which neural inputs (channels, time points, neurons) are most salient for predictions.
• Employs standard deep learning frameworks (PyTorch, TensorFlow) with standard optimization and loss functions, while emphasizing careful cross-validation to prevent overfitting.

Experimental results

Research questions

• RQ1 How do different deep learning architectures (CNNs, LSTMs) improve neural decoding accuracy across diverse neural recording modalities?
• RQ2 To what extent can pretrained deep networks serve as effective priors for decoding high-dimensional outputs such as speech or images from neural signals?
• RQ3 In what ways does transfer learning enhance decoding performance, especially when data is limited or for new subjects?
• RQ4 How can interpretability techniques be applied to understand the transformations learned by neural decoders in neuroscience contexts?
• RQ5 What are the limitations of deep learning in neural decoding when applied to low-SNR or low-sample-size data like fMRI or fNIRS?

Key findings

• Deep learning models, particularly CNNs and LSTMs, outperform linear and shallow methods in neural decoding tasks, even on relatively small datasets.
• Pretrained deep networks significantly improve decoding accuracy for complex outputs such as acoustic speech and visual images by providing strong inductive biases.
• Transfer learning enables effective decoding for new subjects or new outputs with limited data, reducing the need for extensive retraining.
• Interpretability methods such as feature attribution and attention mechanisms can identify which neural inputs (e.g., specific neurons or time points) are most predictive of decoded outputs.
• Despite high performance, the learned representations in decoders are influenced by both neural activity and prior knowledge (e.g., from pretrained models), limiting direct inference about neural information content.
• Overfitting remains a concern in neuroscientific applications, especially with fMRI and fNIRS, where data is sparse and high-dimensional, necessitating careful cross-validation and regularization.