[Paper Review] Dissimilarity learning via Siamese network predicts brain imaging data

Aakash Agrawal|arXiv (Cornell University)|Jul 1, 2019

Neural dynamics and brain function13 references4 citations

TL;DR

This paper proposes a Siamese network-based dissimilarity learning framework that directly trains on neural dissimilarity data to predict brain imaging responses. By modifying the contrastive loss to predict correlation distances between image pairs' features, the model achieves state-of-the-art performance on MEG datasets—particularly for early visual cortex (EVC) and late temporal lobe (IT) responses—outperforming standard classification-based approaches on high signal-to-noise ratio neural data.

ABSTRACT

The advent of deep learning has a profound effect on visual neuroscience. It paved the way for new models to predict neural data. Although deep convolutional neural networks are explicitly trained for categorization, they learn a representation similar to a biological visual system. But categorization is not the only goal of the human visual system. Hence, the representation of a classification algorithm may not completely explain the visual processing stages. Here, I modified the traditional Siamese network loss function (Contrastive loss) to train them directly on neural dissimilarity. This network takes image pair as input and predicts the correlation distance between their output features. For Algonauts challenge, using dissimilarity learning, I fine-tuned the initial layers of Alexnet to predict MEG early response/EVC data and all the layers of VGG-16 to predict MEG late response/IT data. This approach is ideal for datasets with high SNR. Therefore, my model achieved state-of-the-art performance on MEG dataset but not fMRI.

Motivation & Objective

• To develop a deep learning framework that directly models neural dissimilarity rather than relying on classification-based representations.
• To improve prediction of brain imaging responses—particularly MEG data—by training on correlation distances between image pairs.
• To address the limitation of standard CNNs, which optimize for categorization but may not fully capture the functional hierarchy of visual processing.
• To explore whether dissimilarity learning can better align with biological visual processing stages than traditional classification objectives.
• To evaluate performance on MEG data, where signal-to-noise ratio is high, and compare results with fMRI, which has lower SNR.

Proposed method

• Adapts Siamese networks to predict the correlation distance between feature embeddings of image pairs, using a modified contrastive loss function.
• Trains the network end-to-end on image pairs, where the output is the predicted dissimilarity (correlation distance) between their deep features.
• Fine-tunes early layers of AlexNet to predict MEG early response (EVC) data and all layers of VGG-16 to predict MEG late response (IT) data.
• Uses a contrastive loss function reweighted to emphasize learning meaningful dissimilarity in feature space, rather than classification accuracy.
• Optimizes for direct regression of neural dissimilarity, enabling the network to learn representations aligned with actual brain response patterns.
• Applies the trained model to predict neural responses from brain imaging data, particularly focusing on high-SNR MEG recordings.

Experimental results

Research questions

• RQ1 Can a Siamese network trained on neural dissimilarity outperform standard classification-based models in predicting brain imaging responses?
• RQ2 Does direct dissimilarity learning better capture the functional organization of the human visual system than representation learning optimized for classification?
• RQ3 How does the performance of dissimilarity learning vary between high-SNR (MEG) and low-SNR (fMRI) neuroimaging modalities?
• RQ4 To what extent can fine-tuning of pre-trained networks (e.g., AlexNet, VGG-16) improve prediction of specific brain regions (EVC vs. IT) using dissimilarity signals?
• RQ5 Can a modified contrastive loss function effectively train a network to predict correlation distances between neural feature representations?

Key findings

• The proposed dissimilarity learning approach achieved state-of-the-art performance on MEG datasets, particularly for early visual cortex (EVC) and late temporal lobe (IT) responses.
• The model outperformed standard classification-based approaches on high signal-to-noise ratio (SNR) MEG data, indicating its suitability for high-quality neural recordings.
• Fine-tuning the initial layers of AlexNet improved prediction of MEG early response (EVC) data, suggesting alignment with early visual processing stages.
• Fine-tuning all layers of VGG-16 enhanced prediction of MEG late response (IT) data, indicating effective modeling of higher-level visual processing.
• The method did not achieve state-of-the-art performance on fMRI data, likely due to the lower SNR in fMRI, which limits the effectiveness of dissimilarity-based training.
• The modified contrastive loss function successfully enabled the network to learn meaningful dissimilarity representations that correlate with actual neural responses.