[Paper Review] Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?

Subba Reddy Oota, Jashn Arora+4 more|arXiv (Cornell University)|May 3, 2022

Neurobiology of Language and Bilingualism50 references24 citations

TL;DR

This study investigates which task-specific NLP representations from fine-tuned Transformers best predict fMRI brain activity during reading and listening. Using ridge regression on features from 10 NLP tasks, it finds that coreference resolution, NER, and shallow syntax parsing best predict reading-related brain activity, while paraphrase generation, summarization, and natural language inference outperform for listening, revealing distinct neural encoding patterns across modalities.

ABSTRACT

Several popular Transformer based language models have been found to be successful for text-driven brain encoding. However, existing literature leverages only pretrained text Transformer models and has not explored the efficacy of task-specific learned Transformer representations. In this work, we explore transfer learning from representations learned for ten popular natural language processing tasks (two syntactic and eight semantic) for predicting brain responses from two diverse datasets: Pereira (subjects reading sentences from paragraphs) and Narratives (subjects listening to the spoken stories). Encoding models based on task features are used to predict activity in different regions across the whole brain. Features from coreference resolution, NER, and shallow syntax parsing explain greater variance for the reading activity. On the other hand, for the listening activity, tasks such as paraphrase generation, summarization, and natural language inference show better encoding performance. Experiments across all 10 task representations provide the following cognitive insights: (i) language left hemisphere has higher predictive brain activity versus language right hemisphere, (ii) posterior medial cortex, temporo-parieto-occipital junction, dorsal frontal lobe have higher correlation versus early auditory and auditory association cortex, (iii) syntactic and semantic tasks display a good predictive performance across brain regions for reading and listening stimuli resp.

Motivation & Objective

• To identify which task-specific NLP representations from fine-tuned Transformers most effectively predict fMRI brain activity during language processing.
• To compare the predictive power of syntactic and semantic NLP tasks on brain responses for reading versus listening stimuli.
• To uncover cognitive insights about brain network organization by linking task-specific representations to fMRI voxel activity.
• To extend prior work on task-agnostic pretraining by exploring how task-specific fine-tuning influences neural encoding performance.

Proposed method

• Fine-tuned Transformer models on 10 diverse NLP tasks: two syntactic (shallow syntax parsing, word sense disambiguation) and eight semantic (coreference resolution, NER, paraphrase detection, QA, summarization, NLI, sentiment analysis, sentiment analysis).
• Extracted task-specific sentence-level representations from the final layer of each fine-tuned model for input into brain encoding models.
• Applied ridge regression to map task-specific representations to fMRI voxel activity across the whole brain for two datasets: Pereira (reading sentences) and Narratives-Pieman (listening to stories).
• Evaluated model performance using mean absolute error (MAE) and Pearson correlation between predicted and actual fMRI responses.
• Generated brain maps of prediction error (MAE) and correlation across brain regions to visualize task-specific predictive power.
• Conducted sub-ROI analysis on key language network regions (e.g., LMTG, PMC, TPOJ) to identify which brain areas are most responsive to specific NLP tasks.

Experimental results

Research questions

• RQ1 Which NLP tasks produce the most predictive representations for fMRI brain activity during reading of text?
• RQ2 Which NLP tasks yield the highest encoding accuracy for listening to spoken narratives?
• RQ3 How do syntactic and semantic task representations differ in their ability to predict activity in distinct brain regions?
• RQ4 Which brain regions show the strongest correlation with task-specific NLP features, and how do these vary between reading and listening modalities?

Key findings

• Coreference resolution, named entity recognition (NER), and shallow syntax parsing produced the lowest mean absolute error (MAE) in predicting fMRI activity during reading, indicating superior predictive power for this modality.
• For listening tasks, paraphrase detection, summarization, and natural language inference (NLI) showed the lowest MAE values, outperforming other tasks in predicting brain responses to spoken narratives.
• The left hemisphere language network, particularly the posterior temporal gyrus (LPTG) and middle temporal gyrus (LMTG), showed higher predictive accuracy than the right hemisphere for both reading and listening.
• Posterior medial cortex (PMC), temporo-parieto-occipital junction (TPOJ), and dorsal frontal lobe regions exhibited higher Pearson correlation with predicted activity than early auditory and auditory association cortex (EAC/AAC).
• Sub-ROI analysis revealed that LMTG and LPTG were more accurately predicted by coreference resolution features than other language network sub-regions, while PMC and TPOJ regions showed strong correlation with paraphrase detection features.
• The study found that task-specific fine-tuning of Transformers significantly improves brain encoding performance compared to generic pretraining, with distinct optimal tasks for reading versus listening modalities.