[Paper Review] DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders

Shuming Ma, Li Dong+7 more|arXiv (Cornell University)|Jun 25, 2021

Topic ModelingComputer Science31 references51 citations

TL;DR

ΔLM reuses a pretrained multilingual encoder to initialize both encoder and decoder of an encoder-decoder model, and pre-trains it with span corruption and translation span corruption on monolingual and bilingual data to improve multilingual generation and translation.

ABSTRACT

While pretrained encoders have achieved success in various natural language understanding (NLU) tasks, there is a gap between these pretrained encoders and natural language generation (NLG). NLG tasks are often based on the encoder-decoder framework, where the pretrained encoders can only benefit part of it. To reduce this gap, we introduce DeltaLM, a pretrained multilingual encoder-decoder model that regards the decoder as the task layer of off-the-shelf pretrained encoders. Specifically, we augment the pretrained multilingual encoder with a decoder and pre-train it in a self-supervised way. To take advantage of both the large-scale monolingual data and bilingual data, we adopt the span corruption and translation span corruption as the pre-training tasks. Experiments show that DeltaLM outperforms various strong baselines on both natural language generation and translation tasks, including machine translation, abstractive text summarization, data-to-text, and question generation. The code and pretrained models are available at \url{https://aka.ms/deltalm}.

Motivation & Objective

• Motivate bridging the gap between pretrained encoders and natural language generation (NLG) tasks that rely on encoder-decoder frameworks.
• Propose a method to reuse a pretrained multilingual encoder to initialize both encoder and decoder of an encoder-decoder model.
• Leverage large-scale monolingual and bilingual data through specialized pre-training tasks to improve cross-lingual transfer.
• Demonstrate effectiveness on multilingual generation and translation benchmarks across NLG, MT, summarization, data-to-text and question generation.

Proposed method

• Initialize both encoder and decoder of a pretrained multilingual encoder-decoder model from a strong multilingual encoder (InfoXLM).
• Introduce an interleaved Transformer decoder to align decoder structure with the encoder and enable full reuse of pretrained weights.
• Pre-train with span corruption on multilingual data to preserve cross-lingual transferability.
• Enhance cross-lingual transfer by translation-span corruption using bilingual parallel data.
• Use a 6TB multilingual corpus (100 languages) plus 88GB bilingual data (77 languages) with the 360M-parameter base model.
• Fine-tune on downstream tasks with standard optimization and evaluation settings; perform zero-shot transfer experiments with mixed pre-training objectives during fine-tuning.

Experimental results

Research questions

• RQ1 Can reusing a pretrained multilingual encoder to initialize an encoder-decoder model improve NLG and translation tasks?
• RQ2 Does an interleaved decoder enable full utilization of pretrained encoder weights for better cross-lingual generation?
• RQ3 Do span corruption and translation-span corruption tasks effectively leverage monolingual and bilingual data for multilingual NLG and MT?
• RQ4 How does ΔLM perform on multilingual generation, cross-lingual generation, and zero-shot transfer compared to strong baselines?

Key findings

• ΔLM with 360M parameters outperforms XLM and XNLG on XQG-Zh and XGiga-Fr in BLEU, METEOR, and ROUGE-L.
• ΔLM achieves +2.7 average BLEU improvements over multilingual NMT baselines on X→En test sets and +1.3 over En→X.
• ΔLM outperforms mBART and M2M-100 across 10 languages in X→En and En→X directions with fewer parameters.
• On cross-lingual abstractive summarization and data-to-text, ΔLM matches or exceeds baselines such as mBART and mT5 while being more parameter-efficient (360M vs up to 3.7B in some baselines).
• In zero-shot cross-lingual transfer for XGiga, ΔLM significantly outperforms XLM, XLM+MT, and XNLG on French and Chinese test sets.