[Paper Review] Does injecting linguistic structure into language models lead to better alignment with brain recordings?

Mostafa Abdou, Ana Valeria González+3 more|arXiv (Cornell University)|Jan 29, 2021

Topic ModelingComputer Science50 references19 citations

TL;DR

This paper proposes injecting linguistic structure into BERT via attention guidance using syntactic (UD, DM) and semantic (UCCA) formalisms to improve alignment with fMRI brain recordings. It finds that structurally biased models significantly enhance brain decoding performance on one dataset (Wehbe2014), particularly for content words and syntactic tasks, while results are mixed on another (Pereira2018), suggesting structural bias improves neural alignment when task and data conditions are favorable.

ABSTRACT

Neuroscientists evaluate deep neural networks for natural language processing as possible candidate models for how language is processed in the brain. These models are often trained without explicit linguistic supervision, but have been shown to learn some linguistic structure in the absence of such supervision (Manning et al., 2020), potentially questioning the relevance of symbolic linguistic theories in modeling such cognitive processes (Warstadt and Bowman, 2020). We evaluate across two fMRI datasets whether language models align better with brain recordings, if their attention is biased by annotations from syntactic or semantic formalisms. Using structure from dependency or minimal recursion semantic annotations, we find alignments improve significantly for one of the datasets. For another dataset, we see more mixed results. We present an extensive analysis of these results. Our proposed approach enables the evaluation of more targeted hypotheses about the composition of meaning in the brain, expanding the range of possible scientific inferences a neuroscientist could make, and opens up new opportunities for cross-pollination between computational neuroscience and linguistics.

Motivation & Objective

• To investigate whether injecting symbolic linguistic structure into pre-trained language models improves their alignment with human brain activity during language comprehension.
• To evaluate whether attention mechanisms guided by syntactic and semantic formalisms (UD, DM, UCCA) lead to more brain-like representations.
• To assess whether such structural bias enhances performance on both brain decoding and targeted linguistic probing tasks.
• To identify linguistic and data-related confounds affecting model-brain alignment, especially regarding corpus domain and size.
• To enable more targeted neuroscientific hypotheses about the composition of meaning in the brain through structured NLP models.

Proposed method

• Fine-tune BERT using a novel attention-guiding loss that biases self-attention to follow syntactic or semantic annotations from UD (dependency parsing), DM (minimal recursion semantics), and UCCA (semantic dependency formalism).
• Train models on domain-specific text corpora annotated with each formalism, using the annotations to guide attention heads during fine-tuning.
• Extract sentence and word-level representations from both baseline and guided models for comparison with fMRI recordings from two datasets (Wehbe2014 and Pereira2018).
• Perform brain decoding by learning linear mappings from model representations to fMRI activation patterns, evaluating performance via correlation or F1-score.
• Conduct probing tasks to assess syntactic (e.g., subject-verb agreement) and semantic (e.g., semantic tagging) capabilities of the models.
• Analyze performance differences across content vs. function words and evaluate the impact of corpus size, domain, and dependency length as confounds.

Experimental results

Research questions

• RQ1 Does injecting linguistic structure into language models improve their representational alignment with human fMRI brain recordings?
• RQ2 Which linguistic formalism (UD, DM, UCCA) leads to the strongest brain alignment when used to guide attention in BERT?
• RQ3 How does structural bias affect performance on syntactic and semantic probing tasks, and does this correlate with brain alignment?
• RQ4 To what extent do corpus domain and size act as confounding factors in model-brain alignment?
• RQ5 Does the loss of structural information during sentence-level aggregation explain weaker performance on the Pereira2018 dataset?

Key findings

• Models fine-tuned with guided attention based on UD and DM formalisms showed significantly improved brain decoding performance on the Wehbe2014 dataset, particularly at the word level.
• The improvement was most pronounced for content words, as the performance drop in domain-fine-tuned baselines (UD and DM) was largely driven by reduced decoding accuracy on content words.
• For the Pereira2018 dataset, improvements were more modest at the sentence level, but when mean-pooled representations were used, the advantage of guided models disappeared, suggesting aggregation may mask structural gains.
• Models with attention guided by DM and UD consistently outperformed those guided by UCCA in brain alignment across both datasets, though this may be confounded by corpus domain and size.
• Higher decoding scores on syntactic probing tasks (e.g., subject-verb agreement) were consistently observed for models that aligned best with brain activity, indicating shared structural sensitivity.
• The study reveals that corpus domain and size are critical confounds: the rank of decoding performance for guided models correlated with training corpus size (DM > UD > UCCA), suggesting data characteristics significantly influence structural bias effectiveness.