[Paper Review] Zero-Shot Cross-Lingual Dependency Parsing through Contextual Embedding Transformation

Haoran Xu, Philipp Koehn|arXiv (Cornell University)|Mar 3, 2021

Topic Modeling26 references4 citations

TL;DR

This paper proposes a zero-shot cross-lingual dependency parsing method using sense-level contextual embedding transformation, leveraging silver-aligned parallel sentences and iterative normalization to mitigate anisotropy. It outperforms state-of-the-art methods on Universal Dependencies 2.6, achieving significant gains in both UAS and LAS by mapping multi-sense word representations via k-means clustering and orthogonal Procrustes alignment.

ABSTRACT

Linear embedding transformation has been shown to be effective for zero-shot cross-lingual transfer tasks and achieve surprisingly promising results. However, cross-lingual embedding space mapping is usually studied in static word-level embeddings, where a space transformation is derived by aligning representations of translation pairs that are referred from dictionaries. We move further from this line and investigate a contextual embedding alignment approach which is sense-level and dictionary-free. To enhance the quality of the mapping, we also provide a deep view of properties of contextual embeddings, i.e., anisotropy problem and its solution. Experiments on zero-shot dependency parsing through the concept-shared space built by our embedding transformation substantially outperform state-of-the-art methods using multilingual embeddings.

Motivation & Objective

• To improve zero-shot cross-lingual dependency parsing by moving beyond static word-level embeddings to sense-level contextual representations.
• To eliminate reliance on gold dictionaries by using silver-aligned token pairs from parallel corpora.
• To address the anisotropy problem in contextual embeddings that degrades zero-shot transfer performance.
• To enhance mapping precision by clustering multi-sense word embeddings into sense-level representations before transformation.
• To achieve state-of-the-art performance in zero-shot dependency parsing using multilingual BERT and linear transformation.

Proposed method

• Uses Fast Align to extract silver-aligned token pairs from parallel monolingual corpora.
• Applies monolingual BERT models to obtain contextual embeddings for both source and target languages.
• Applies k-means clustering to target-side contextual embeddings per word type to derive sense-level representations.
• Uses aligned source-side embeddings to simultaneously form sense-level representations on the source side.
• Applies iterative normalization (IN) to reduce anisotropy in contextual embeddings before transformation.
• Derives a linear transformation matrix via Procrustes rotation on sense-level embedding pairs to align source and target spaces.

Experimental results

Research questions

• RQ1 Can sense-level contextual embedding alignment outperform word-level alignment in zero-shot cross-lingual dependency parsing?
• RQ2 Does a dictionary-free approach using silver-aligned parallel data achieve better performance than methods relying on gold dictionaries?
• RQ3 To what extent does mitigating anisotropy in contextual embeddings improve zero-shot transfer performance?
• RQ4 Can k-means clustering of contextual embeddings effectively separate multi-sense word representations for improved mapping?
• RQ5 How does the proposed method compare to state-of-the-art approaches using multilingual fastText and mBERT on standard benchmarks?

Key findings

• The proposed sense-level embedding transformation method achieves higher UAS and LAS scores than word-level mapping across all zero-shot cross-lingual parsing evaluations.
• The method outperforms state-of-the-art approaches using multilingual fastText and mBERT on Universal Dependencies 2.6.
• Iterative normalization significantly reduces the anisotropy of contextual embeddings, improving the quality of the linear transformation.
• Sense-level clustering improves mapping precision by separating different word senses into distinct vector clusters.
• The dictionary-free approach using Fast Align and parallel corpora achieves strong performance without requiring gold bilingual lexicons.
• The method demonstrates consistent gains across diverse language pairs, indicating robust generalization in zero-shot settings.