[Paper Review] Subword Sampling for Low Resource Word Alignment

Ehsaneddin Asgari, Masoud Jalili Sabet+3 more|arXiv (Cornell University)|Dec 21, 2020

Natural Language Processing Techniques27 references4 citations

TL;DR

This paper proposes a subword sampling method that improves word-level alignment in low-resource language pairs by aggregating multiple subword segmentations, using an iterative Bayesian optimization framework to select optimal subwords. It achieves F1 scores comparable to 100K+ sentence word-level models using only 5,000 parallel sentences across six language pairs, with transferable hyperparameters enabling zero-shot improvement on unseen languages.

ABSTRACT

Annotation projection is an important area in NLP that can greatly contribute to creating language resources for low-resource languages. Word alignment plays a key role in this setting. However, most of the existing word alignment methods are designed for a high resource setting in machine translation where millions of parallel sentences are available. This amount reduces to a few thousands of sentences when dealing with low-resource languages failing the existing established IBM models. In this paper, we propose subword sampling-based alignment of text units. This method's hypothesis is that the aggregation of different granularities of text for certain language pairs can help word-level alignment. For certain languages for which gold-standard alignments exist, we propose an iterative Bayesian optimization framework to optimize selecting possible subwords from the space of possible subword representations of the source and target sentences. We show that the subword sampling method consistently outperforms word-level alignment on six language pairs: English-German, English-French, English-Romanian, English-Persian, English-Hindi, and English-Inuktitut. In addition, we show that the hyperparameters learned for certain language pairs can be applied to other languages at no supervision and consistently improve the alignment results. We observe that using $5K$ parallel sentences together with our proposed subword sampling approach, we obtain similar F1 scores to the use of $100K$'s of parallel sentences in existing word-level fast-align/eflomal alignment methods.

Motivation & Objective

• To address the challenge of unreliable word alignment in low-resource language pairs where parallel corpora contain only a few thousand sentences.
• To improve annotation projection for low-resource languages by enabling accurate mapping of linguistic units across languages.
• To develop a method that outperforms standard word-level alignment in low-resource settings using subword-level granularity.
• To enable transferability of learned hyperparameters across language pairs without additional supervision.

Proposed method

• The method samples from the space of possible BPE subword segmentations for source and target sentences to explore diverse text granularities.
• An iterative Bayesian optimization framework is used to select the most effective subword candidates based on gold-standard alignments for languages with available reference data.
• Subword representations are generated via byte-pair encoding (BPE), and multiple segmentations are considered to increase alignment robustness.
• The framework optimizes hyperparameters for subword sampling, such as subword merging thresholds, to maximize F1 score on held-out gold-standard alignments.
• The method is trained on gold-standard alignments for specific language pairs and then applied to other pairs with no further supervision.
• It leverages the insight that combining multiple subword granularities improves alignment performance, especially when training data is scarce.

Experimental results

Research questions

• RQ1 Can subword sampling improve word-level alignment performance in low-resource language pairs with limited parallel sentences?
• RQ2 How does the aggregation of multiple subword segmentations enhance alignment accuracy compared to fixed word-level alignment?
• RQ3 Can hyperparameters learned on one language pair be effectively transferred to other language pairs without supervision?
• RQ4 To what extent can subword sampling reduce the required parallel sentence count to achieve performance comparable to high-resource word-level models?
• RQ5 Does the method maintain consistent improvements across diverse language pairs, including typologically distant ones?

Key findings

• The subword sampling method consistently outperforms word-level alignment on six language pairs: English-German, English-French, English-Romanian, English-Persian, English-Hindi, and English-Inuktitut.
• Using only 5,000 parallel sentences, the method achieves F1 scores comparable to those obtained with 100,000+ sentences using standard word-level alignment methods.
• For English-Inuktitut, the method matched the F1 score of a 340K-sentence word-level model using just 5,000 sentences.
• For English-Persian, the method matched the F1 score of a 600K-sentence word-level model with only 5,000 sentences.
• Hyperparameters learned on one language pair were successfully transferred to other language pairs with no supervision, consistently improving alignment results.
• The method demonstrates strong generalization, enabling high-quality alignment even in mid-resource settings with 50K–600K sentences.