[Paper Review] Integrating Linguistic Theory and Neural Language Models

Bai Li|arXiv (Cornell University)|Jul 20, 2022

Natural Language Processing Techniques4 citations

TL;DR

This dissertation demonstrates the mutual benefit between linguistic theory and neural language models by using multilingual BERT to measure semantic distances in word class flexibility, introducing a Gaussian-based surprisal method to detect morphosyntactic anomalies earlier than semantic ones, and adapting psycholinguistic studies to show that language models encode argument structure constructions. The key contribution is establishing systematic, theory-informed probing methods that reveal how neural models internalize linguistic phenomena.

ABSTRACT

Transformer-based language models have recently achieved remarkable results in many natural language tasks. However, performance on leaderboards is generally achieved by leveraging massive amounts of training data, and rarely by encoding explicit linguistic knowledge into neural models. This has led many to question the relevance of linguistics for modern natural language processing. In this dissertation, I present several case studies to illustrate how theoretical linguistics and neural language models are still relevant to each other. First, language models are useful to linguists by providing an objective tool to measure semantic distance, which is difficult to do using traditional methods. On the other hand, linguistic theory contributes to language modelling research by providing frameworks and sources of data to probe our language models for specific aspects of language understanding. This thesis contributes three studies that explore different aspects of the syntax-semantics interface in language models. In the first part of my thesis, I apply language models to the problem of word class flexibility. Using mBERT as a source of semantic distance measurements, I present evidence in favour of analyzing word class flexibility as a directional process. In the second part of my thesis, I propose a method to measure surprisal at intermediate layers of language models. My experiments show that sentences containing morphosyntactic anomalies trigger surprisals earlier in language models than semantic and commonsense anomalies. Finally, in the third part of my thesis, I adapt several psycholinguistic studies to show that language models contain knowledge of argument structure constructions. In summary, my thesis develops new connections between natural language processing, linguistic theory, and psycholinguistics to provide fresh perspectives for the interpretation of language models.

Motivation & Objective

• To investigate whether neural language models can serve as objective tools for measuring semantic distance in linguistic phenomena such as word class flexibility.
• To develop a method for measuring surprisal at intermediate layers of language models using Gaussian density estimation to detect linguistic anomalies.
• To adapt psycholinguistic experiments to test whether language models encode knowledge of argument structure constructions from construction grammar theory.
• To demonstrate that linguistic theory can guide probing methods, improving interpretability and relevance of neural language model analysis.
• To establish a framework for integrating theoretical linguistics, psycholinguistics, and NLP through systematic probing of language models.

Proposed method

• Used mBERT to compute semantic similarity scores across multilingual word pairs to assess word class flexibility as a directional process.
• Proposed a Gaussian-based density estimation method to compute surprisal at intermediate transformer layers, enabling layerwise anomaly detection.
• Applied the surprisal method to three types of linguistic anomalies: morphosyntactic, semantic, and commonsense, using curated datasets.
• Adapted psycholinguistic paradigms—specifically N400 and priming studies—into probing tasks to evaluate language model responses to argument structure constructions.
• Conducted layerwise probing of contextual embeddings to analyze how linguistic features emerge during the forward pass of transformer models.
• Used ablation studies and model comparisons (e.g., RoBERTa vs. XLNet) to assess sensitivity to architectural differences and data distribution.

Experimental results

Research questions

• RQ1 Can language models like mBERT provide objective, cross-linguistically consistent measures of semantic distance that support the directional analysis of word class flexibility?
• RQ2 At which layer do language models first detect morphosyntactic anomalies compared to semantic or commonsense anomalies, and what does this reveal about their internal processing?
• RQ3 Do language models exhibit evidence of argument structure construction knowledge comparable to human psycholinguistic findings?
• RQ4 How do architectural differences in language models (e.g., RoBERTa vs. XLNet) affect their sensitivity to linguistic anomaly types?
• RQ5 Can psycholinguistic paradigms be effectively adapted to probe neural language models for specific linguistic phenomena?

Key findings

• mBERT successfully identified directional patterns in word class flexibility, supporting the hypothesis that semantic shifts in conversion are asymmetric and directional across languages.
• Morphosyntactic anomalies triggered higher surprisal earlier in the network (e.g., in lower layers) compared to semantic or commonsense anomalies, indicating early sensitivity to syntactic violations.
• Language models showed robust responses to argument structure construction probes, replicating key psycholinguistic findings such as priming effects and N400-like responses.
• The Gaussian-based surprisal method detected anomalies earlier and more consistently than standard masked language modeling, particularly for morphosyntactic errors.
• Architectural differences significantly impacted anomaly detection: RoBERTa showed clearer distinctions between anomaly types than XLNet, suggesting model architecture influences linguistic sensitivity.
• Probing results were robust across multiple languages and datasets, indicating that linguistic knowledge is systematically encoded in contextual embeddings, not just statistical patterns.