[Paper Review] Simple, Scalable Adaptation for Neural Machine Translation

Ankur Bapna, Naveen Arivazhagan+1 more|arXiv (Cornell University)|Sep 18, 2019

Natural Language Processing Techniques37 references13 citations

TL;DR

This paper proposes lightweight adapter layers inserted into pre-trained NMT models to enable efficient, parameter-efficient adaptation for multiple domains and languages. By fine-tuning only these small, task-specific adapters while keeping the main model frozen, the approach achieves performance on par with full fine-tuning across domain adaptation and massively multilingual translation (103 languages), significantly reducing parameter overhead and eliminating the need for hyperparameter tuning.

ABSTRACT

Fine-tuning pre-trained Neural Machine Translation (NMT) models is the dominant approach for adapting to new languages and domains. However, fine-tuning requires adapting and maintaining a separate model for each target task. We propose a simple yet efficient approach for adaptation in NMT. Our proposed approach consists of injecting tiny task specific adapter layers into a pre-trained model. These lightweight adapters, with just a small fraction of the original model size, adapt the model to multiple individual tasks simultaneously. We evaluate our approach on two tasks: (i) Domain Adaptation and (ii) Massively Multilingual NMT. Experiments on domain adaptation demonstrate that our proposed approach is on par with full fine-tuning on various domains, dataset sizes and model capacities. On a massively multilingual dataset of 103 languages, our adaptation approach bridges the gap between individual bilingual models and one massively multilingual model for most language pairs, paving the way towards universal machine translation.

Motivation & Objective

• To address the inefficiency and scalability issues of full fine-tuning in NMT adaptation for multiple domains and low-resource languages.
• To reduce the need for maintaining separate models per language or domain by enabling a single model to adapt to multiple tasks simultaneously.
• To develop a parameter-efficient, hyper-parameter-insensitive method that avoids catastrophic forgetting during adaptation.
• To improve translation quality on low-resource languages in a massively multilingual setting without degrading performance on high-resource language pairs.
• To enable flexible adaptation by adjusting adapter capacity based on task complexity and data size.

Proposed method

• Inject small, task-specific adapter layers between the layers of a pre-trained Transformer NMT model, keeping the main model parameters frozen.
• Use residual adapter blocks with a bottleneck structure (down-projection, ReLU, up-projection) to control parameter efficiency and capacity.
• Fine-tune only the adapter parameters on target domain or language-specific data, while the base model remains fixed.
• Train a single global multilingual model on 103 languages, then inject and fine-tune language-pair-specific adapters for each language pair.
• Adjust adapter bottleneck dimension (b=2048 or b=4096) to match task complexity and data size, with larger adapters used for higher-resource languages.
• Combine all fine-tuned adapters into a single model after individual fine-tuning stages to support multi-task inference.

Experimental results

Research questions

• RQ1 Can a single pre-trained NMT model be efficiently adapted to multiple domains and languages using only small, task-specific adapter layers?
• RQ2 Does adapter-based adaptation achieve performance comparable to full fine-tuning while reducing parameter overhead and training complexity?
• RQ3 Can adapters effectively bridge the performance gap between bilingual models and a single multilingual model, especially for low-resource language pairs?
• RQ4 How does adapter capacity (bottleneck size) affect performance across different data sizes and language resource levels?
• RQ5 Can adapters be used to simultaneously adapt a model to multiple domains and languages without catastrophic forgetting?

Key findings

• Adapter-based adaptation achieves translation performance on par with full fine-tuning across various domains, dataset sizes, and model capacities, with only a fraction of the parameters updated.
• On a 103-language multilingual NMT setup, adapters significantly close the performance gap between multilingual and bilingual models, especially for low-resource language pairs.
• High-resource language pairs show further performance gains when adapter size is increased (e.g., b=4096), demonstrating the flexibility of adapter capacity scaling.
• The approach reduces memory and training cost by avoiding separate model training per language or domain, enabling scalable, multi-task adaptation.
• Despite improvements, a minor performance regression remains for high-resource languages translating into English compared to bilingual baselines, suggesting room for further optimization.
• Adapter fine-tuning converges faster than training from scratch, typically within 20k–50k steps depending on corpus size, due to the small number of trainable parameters.