[论文解读] Identifying Necessary Elements for BERT's Multilinguality

Philipp Dufter, Hinrich Schütze|arXiv (Cornell University)|May 1, 2020

Natural Language Processing Techniques被引用 10

一句话总结

本文 identifies four architectural and two linguistic factors essential for BERT's multilinguality: shared position embeddings, shared special tokens, random token masking, parameter efficiency, consistent word order, and comparable training data. Using a lightweight, synthetic-data-based setup, the authors demonstrate that these elements enable effective zero-shot cross-lingual transfer, with findings validated on XNLI across three languages, showing that multilinguality is enhanced not by explicit cross-lingual signals but by structural and representational alignment in the model's design.

ABSTRACT

It has been shown that multilingual BERT (mBERT) yields high quality multilingual representations and enables effective zero-shot transfer. This is surprising given that mBERT does not use any crosslingual signal during training. While recent literature has studied this phenomenon, the reasons for the multilinguality are still somewhat obscure. We aim to identify architectural properties of BERT and linguistic properties of languages that are necessary for BERT to become multilingual. To allow for fast experimentation we propose an efficient setup with small BERT models trained on a mix of synthetic and natural data. Overall, we identify four architectural and two linguistic elements that influence multilinguality. Based on our insights, we experiment with a multilingual pretraining setup that modifies the masking strategy using VecMap, i.e., unsupervised embedding alignment. Experiments on XNLI with three languages indicate that our findings transfer from our small setup to larger scale settings.

研究动机与目标

• To identify the architectural and linguistic components that enable multilingual BERT (mBERT) to generalize across languages despite no explicit cross-lingual training signals.
• To develop a fast, efficient experimental setup using synthetic and natural data to isolate and test factors influencing multilinguality.
• To investigate whether overparameterization or structural differences (e.g., inverted word order) impair multilingual representation learning.
• To validate findings from small-scale experiments on a larger, real-world setting using Wikipedia and XNLI.
• To propose a simple, unsupervised modification to improve multilinguality by aligning masking strategy via VecMap, without additional supervision or complex loss functions.

提出的方法

• Propose a minimal BERT setup with small models trained on synthetic (fake-english) and natural (easy-to-read bible) data to enable rapid experimentation.
• Generate fake-english by shifting token indices post-tokenization and prefixing with '::' to preserve linguistic structure while altering representation.
• Use a shared subword vocabulary and controlled data mix to isolate architectural and linguistic variables.
• Train models with controlled architectural modifications: removing shared special tokens (no-special), removing shared position embeddings (no-pos), replacing masked tokens with random tokens (no-random), and shifting token indices (shift-special).
• Apply VecMap to align word embeddings across languages and modify the masking strategy to enhance cross-lingual alignment.
• Evaluate multilinguality on XNLI using sentence pair classification across three languages, measuring zero-shot transfer performance.

实验结果

研究问题

• RQ1 Which architectural components of BERT are necessary for multilingual representation learning?
• RQ2 How do linguistic properties such as word order and structural similarity influence multilingual generalization in BERT?
• RQ3 Does overparameterization or training duration conflict with multilinguality, and if so, how?
• RQ4 Can multilinguality be enhanced through unsupervised alignment of masked tokens without additional supervision?
• RQ5 Do findings from a small-scale synthetic setup generalize to larger, real-world multilingual settings?

主要发现

• Shared position embeddings and shared special tokens are essential for multilingual representation alignment; removing them severely degrades multilinguality.
• Replacing masked tokens with random tokens (rather than [MASK]) significantly improves multilingual performance, suggesting dynamic masking aids cross-lingual generalization.
• Models with inverted word order in one language lose multilingual capability, indicating that consistent syntactic structure is crucial for cross-lingual transfer.
• Overparameterization and extended training duration are linked to reduced multilinguality, suggesting a trade-off between generalization and multilingual capacity.
• The proposed VecMap-based masking strategy improves multilingual performance on XNLI, demonstrating that unsupervised alignment can enhance multilinguality without additional supervision.
• Findings from the small-scale setup transfer to larger-scale settings: the modified model achieves strong zero-shot performance on XNLI across three languages, confirming the robustness of the insights.