[Paper Review] A Systematic Analysis of Morphological Content in BERT Models for Multiple Languages

Daniel Edmiston|arXiv (Cornell University)|Apr 6, 2020

Topic Modeling36 references30 citations

TL;DR

The paper probes how BERT-style models encode morphological features across five European languages, showing convex, feature-value separability in embeddings, contextual disambiguation benefits, and targeted attention related to subject-verb agreement.

ABSTRACT

This work describes experiments which probe the hidden representations of several BERT-style models for morphological content. The goal is to examine the extent to which discrete linguistic structure, in the form of morphological features and feature values, presents itself in the vector representations and attention distributions of pre-trained language models for five European languages. The experiments contained herein show that (i) Transformer architectures largely partition their embedding space into convex sub-regions highly correlated with morphological feature value, (ii) the contextualized nature of transformer embeddings allows models to distinguish ambiguous morphological forms in many, but not all cases, and (iii) very specific attention head/layer combinations appear to hone in on subject-verb agreement.

Motivation & Objective

• Assess whether discrete morphological features and their values are reflected in BERT-style models' hidden representations and attention distributions across multiple languages.
• Determine whether embedding spaces are partitioned into convex regions corresponding to feature values.
• Evaluate whether contextualized embeddings aid disambiguation of morphologically ambiguous forms.
• Investigate whether attention patterns reveal agreement relations like subject-verb concord in various languages.

Proposed method

• Conduct two main experimental paradigms on multilingual BERT-base models: (a) supervised classification of morphological feature values from hidden-layer embeddings using k-means, linear, and non-linear classifiers across 12 layers; (b) analysis of self-attention distributions via a Pearson chi-squared-based metric to detect agreement patterns in attention heads and layers.
• Use data from UD Treebanks and UD-compatible lexicons to sample 750 examples per feature value (with some exceptions due to data availability) for training/testing split 85/15.
• Examine features: Case, Gender, Mood, Number, Person, Tense, Verb Form across English, French, German, Russian, Spanish; employ models pre-trained for each language (BERT-base variants).
• Assess performance as a function of feature complexity (number of values, ambiguity) and layer depth to identify where morphological information is most salient.

Experimental results

Research questions

• RQ1 Do BERT-style models encode morphological feature values in their hidden representations across multiple languages?
• RQ2 Is the embedding space partitioned into convex regions corresponding to feature values, enabling linear separability?
• RQ3 Does contextualization help disambiguate morphologically ambiguous forms, and what are its limitations?
• RQ4 Can attention distributions reveal agreement relations (subject-verb) in BERT models, and are such signals localized to specific heads/layers?
• RQ5 How do ambiguity and the number of feature values affect morphological classification performance across languages?

Key findings

• Linear classifiers achieve high F1 scores (often above 0.9) indicating convex, linearly separable regions by feature value in embeddings.
• K-means clustering performances largely near random baselines, suggesting supervision substantially aids extraction of morphological information.
• Mid to late layers (varying by language) best support feature-value classification, with German and Russian showing strongest layer-based trends.
• Contextualized embeddings improve disambiguation of morphologically ambiguous forms but do not fully solve the ambiguity problem, especially for highly syncretic forms.
• Attention analyses show that a minority of head-layer combinations capture the agree relation with statistically significant average Agree-scores, concentrated in early-to-middle layers across languages.
• Languages with simpler morphology (English, French, Spanish) show robust performance across layers, while German and Russian show clearer layer-wise peaks for certain tasks.
• Overall, morphological information in BERT-style models tends to be most salient in middle layers and can be localized to specific attention-head/layer pairs for agreement phenomena.