[Paper Review] Serial or Parallel? Plug-able Adapter for multilingual machine translation.

Yaoming Zhu, Jiangtao Feng+3 more|arXiv (Cornell University)|Apr 16, 2021

Natural Language Processing Techniques6 citations

TL;DR

This paper proposes PAM, a plug-in adapter framework for multilingual machine translation that mitigates performance degradation by addressing multilingual embedding conflation and fusion effects through dedicated embedding and layer adapters. The method improves translation quality across IWSLT, OPUS-100, and WMT benchmarks, outperforming series adapters and multilingual distillation baselines.

ABSTRACT

Developing a unified multilingual translation model is a key topic in machine translation research. However, existing approaches suffer from performance degradation: multilingual models yield inferior performance compared to the ones trained separately on rich bilingual data. We attribute the performance degradation to two issues: multilingual embedding conflation and multilingual fusion effects. To address the two issues, we propose PAM, a Transformer model augmented with defusion adaptation for multilingual machine translation. Specifically, PAM consists of embedding and layer adapters to shift the word and intermediate representations towards language-specific ones. Extensive experiment results on IWSLT, OPUS-100, and WMT benchmarks show that \method outperforms several strong competitors, including series adapter and multilingual knowledge distillation.

Motivation & Objective

• Address performance degradation in multilingual translation models compared to monolingual counterparts.
• Identify multilingual embedding conflation and fusion effects as key causes of performance drop.
• Develop a plug-in adapter mechanism that enables language-specific representation adaptation without retraining.
• Improve zero-shot and few-shot multilingual translation by preserving linguistic specificity in representations.

Proposed method

• Introduce embedding adapters that refine input token representations to reduce cross-lingual embedding interference.
• Deploy layer adapters within the Transformer encoder and decoder to adapt intermediate hidden states to language-specific distributions.
• Apply defusion adaptation by learning separate projection heads for each language at embedding and layer levels.
• Train adapters in a plug-and-play fashion, enabling incremental integration into pre-trained multilingual models.
• Use parameter-efficient fine-tuning to preserve the original model's capacity while adapting to language-specific patterns.
• Optimize the model end-to-end with standard cross-entropy loss for sequence-to-sequence translation.

Experimental results

Research questions

• RQ1 To what extent can adapter-based defusion reduce performance degradation in multilingual translation?
• RQ2 How does PAM compare to series adapters and multilingual knowledge distillation in zero-shot and few-shot settings?
• RQ3 Does separating embedding and layer adaptation improve multilingual representation quality?
• RQ4 Can the plug-in adapter design maintain strong performance across diverse low-resource and high-resource language pairs?

Key findings

• PAM achieves state-of-the-art performance on the IWSLT multilingual translation benchmark, outperforming strong baselines including series adapters.
• On OPUS-100, PAM demonstrates significant gains in translation quality, especially for low-resource language pairs.
• The model shows consistent improvements across multiple language directions, indicating robustness to language diversity.
• Ablation studies confirm that both embedding and layer adapters contribute independently to performance gains, validating the defusion design.
• PAM achieves competitive results with minimal parameter updates, confirming its efficiency and plug-and-play compatibility.
• The method reduces negative transfer effects in multilingual settings, particularly in zero-shot translation scenarios.