[Paper Review] Language-Family Adapters for Low-Resource Multilingual Neural Machine Translation

Alexandra Chronopoulou, Dario Stojanovski+1 more|arXiv (Cornell University)|Sep 30, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes language-family adapters for low-resource multilingual neural machine translation, training lightweight adapters on groups of linguistically related languages (e.g., Balto-Slavic, Indo-Iranian) while keeping the mBART-50 backbone frozen. The method achieves +1.0 BLEU over language-pair adapters and +2.7 BLEU over language-agnostic adapters on 16 low-resource language pairs, with under 20% of the parameters, and improves zero-shot transfer to unseen languages by leveraging cross-lingual similarity.

ABSTRACT

Large multilingual models trained with self-supervision achieve state-of-the-art results in a wide range of natural language processing tasks. Self-supervised pretrained models are often fine-tuned on parallel data from one or multiple language pairs for machine translation. Multilingual fine-tuning improves performance on low-resource languages but requires modifying the entire model and can be prohibitively expensive. Training a new adapter on each language pair or training a single adapter on all language pairs without updating the pretrained model has been proposed as a parameter-efficient alternative. However, the former does not permit any sharing between languages, while the latter shares parameters for all languages and is susceptible to negative interference. In this paper, we propose training language-family adapters on top of mBART-50 to facilitate cross-lingual transfer. Our approach outperforms related baselines, yielding higher translation scores on average when translating from English to 17 different low-resource languages. We also show that language-family adapters provide an effective method to translate to languages unseen during pretraining.

Motivation & Objective

• To address the challenge of low-resource multilingual neural machine translation (NMT) by improving parameter-efficient fine-tuning beyond existing adapter methods.
• To reduce negative interference in parameter-efficient adaptation by grouping languages based on linguistic similarity rather than training a single shared adapter for all languages.
• To enable effective zero-shot translation to languages not seen during pretraining by leveraging shared cross-lingual representations within language families.
• To evaluate whether linguistic knowledge-based grouping outperforms representation-based clustering (e.g., GMM) for adapter training in multilingual NMT.
• To assess the impact of inserting embedding-layer adapters to encode lexical-level information in the Transformer architecture.

Proposed method

• Training language-family adapters on top of the frozen mBART-50 model, where each adapter is trained on parallel monolingual data from a specific language family (e.g., Indo-Iranian, Austronesian).
• Grouping languages using linguistic knowledge bases (e.g., Glottolog) to form families based on shared grammatical and phonological properties, rather than model-derived representations.
• Inserting adapter modules in the encoder and decoder layers of the Transformer, with a separate adapter per language family, to enable parameter-efficient fine-tuning.
• Applying a two-stage training process: first, training the language-family adapters on parallel data from their respective language families; second, evaluating on both seen and unseen languages.
• Using a Gaussian Mixture Model (GMM) on multilingual sentence representations to cluster languages for comparison with linguistic-based grouping.
• Conducting ablation studies to evaluate the contribution of embedding-layer adapters and the impact of different clustering strategies on translation performance.

Experimental results

Research questions

• RQ1 Does training adapters on linguistically grouped language families improve translation performance on low-resource languages compared to language-pair or language-agnostic adapters?
• RQ2 Can language-family adapters effectively transfer knowledge to languages unseen during mBART-50 pretraining?
• RQ3 How does linguistic-based grouping compare to representation-based clustering (e.g., GMM) in terms of translation quality and robustness?
• RQ4 What is the contribution of inserting embedding-layer adapters to encode lexical-level information in the model?
• RQ5 To what extent does the proposed method reduce negative interference compared to a single shared adapter for all languages?

Key findings

• Language-family adapters achieve a +1.0 BLEU improvement over language-pair adapters and +2.7 BLEU over language-agnostic adapters on 16 low-resource language pairs from OPUS-100.
• The method outperforms both baselines across all language families, with gains exceeding +2.0 BLEU on Austronesian and Indo-Iranian families.
• On unseen languages not in mBART-50’s pretraining data, language-family adapters significantly outperform both baselines, demonstrating effective zero-shot transfer.
• The language-agnostic adapter baseline performs worse than both the language-family and language-pair adapters, especially on unseen languages, due to negative interference from unrelated language clusters.
• Inserting embedding-layer adapters improves performance, as shown by ablation studies, indicating that lexical-level information is beneficial for low-resource translation.
• Linguistic knowledge-based grouping consistently outperforms GMM-based clustering on multilingual sentence representations, confirming the value of domain-specific linguistic structure in adapter design.