[Paper Review] Capturing cross-session neural population variability through self-supervised identification of consistent neuron ensembles

Justin Jude, Matthew G. Perich+2 more|arXiv (Cornell University)|May 19, 2022

Neural dynamics and brain function4 citations

TL;DR

This paper introduces CAPTIVATE, a self-supervised deep learning framework that identifies consistent neuron ensembles across neural recording sessions to maintain high-accuracy behaviour decoding despite inter-session neural variability. By training a recurrent neural network to predict original neuron positions from perturbed data mimicking real session-to-session changes—such as neuron loss, replacement, and probe shifts—the model enables a sequential autoencoder to reconstruct neural dynamics and decode behaviour with state-of-the-art performance on unseen sessions up to 8 days later, using only a single training session.

ABSTRACT

Decoding stimuli or behaviour from recorded neural activity is a common approach to interrogate brain function in research, and an essential part of brain-computer and brain-machine interfaces. Reliable decoding even from small neural populations is possible because high dimensional neural population activity typically occupies low dimensional manifolds that are discoverable with suitable latent variable models. Over time however, drifts in activity of individual neurons and instabilities in neural recording devices can be substantial, making stable decoding over days and weeks impractical. While this drift cannot be predicted on an individual neuron level, population level variations over consecutive recording sessions such as differing sets of neurons and varying permutations of consistent neurons in recorded data may be learnable when the underlying manifold is stable over time. Classification of consistent versus unfamiliar neurons across sessions and accounting for deviations in the order of consistent recording neurons in recording datasets over sessions of recordings may then maintain decoding performance. In this work we show that self-supervised training of a deep neural network can be used to compensate for this inter-session variability. As a result, a sequential autoencoding model can maintain state-of-the-art behaviour decoding performance for completely unseen recording sessions several days into the future. Our approach only requires a single recording session for training the model, and is a step towards reliable, recalibration-free brain computer interfaces.

Motivation & Objective

• To address the challenge of inter-session neural variability in brain-computer interfaces, where decoder performance degrades over time due to neuron drift and recording instabilities.
• To develop a method that enables reliable behaviour decoding from completely unseen recording sessions without requiring retraining or recalibration.
• To identify and leverage consistent neuron ensembles across sessions despite changes in neuron identity, order, and activity patterns.
• To demonstrate that self-supervised learning on a single session can generalize to future sessions with high decoding accuracy.

Proposed method

• A self-supervised recurrent neural network (RNN) is trained to predict the original neuron positions in a recording session from multiple perturbed versions of the same trial, simulating real inter-session variability.
• Perturbations include adding, removing, replacing, deleting, jittering, and reordering neurons, reflecting key sources of neural ensemble variability.
• The model uses a joint training scheme where the RNN's output (neuron location embedding) is fed into a Latent Factor Analysis via Dynamical Systems (LFADS) autoencoder to reconstruct original neural activity.
• The LFADS encoder is conditioned on the RNN's neuron location embedding, enabling it to generate behaviourally relevant latent variables even without explicit behaviour supervision.
• An alignment loss is applied to ensure that the reconstructed latent variables from unseen sessions align with those from the training session, improving decoding performance.
• The model is trained on a single session and tested on subsequent sessions, with performance evaluated via behaviour decoding accuracy (R²) on unseen trials.

Experimental results

Research questions

• RQ1 Can a self-supervised model identify consistent neuron ensembles across neural recording sessions despite substantial inter-session variability?
• RQ2 To what extent can a single-session-trained model maintain high decoding accuracy for behaviour prediction on unseen sessions up to several days later?
• RQ3 Which types of inter-session perturbations—such as neuron replacement, deletion, or probe shift—most significantly degrade decoding performance?
• RQ4 How does the model's performance scale with the number of persistent neurons across sessions?

Key findings

• CAPTIVATE achieves a mean decoding performance of R² = 0.84 (±0.02) on Monkey C and R² = 0.86 (±0.03) on Monkey M when testing on the first unseen session after training on day 0.
• Ablation studies show that neuron replacement and reordering perturbations have the highest impact on decoding performance, with removal and deletion also contributing significantly.
• The model maintains high decoding accuracy for up to 8 days, after which performance gradually declines due to increasing variability beyond the model's compensation capacity.
• Removing the alignment loss reduces performance only slightly (to R² = 0.82 on Monkey C and 0.85 on Monkey M), indicating that self-supervised alignment is effective without explicit supervision.
• The model generalizes well to sessions with fewer neurons, and in some cases even outperforms models trained on larger neuron counts, demonstrating robustness to variable recording quality.
• Performance is symmetrical when training on day 0 or day 11, indicating that the model can generalize both forward and backward in time, suggesting strong temporal invariance.