[Paper Review] Incorporating BERT into Parallel Sequence Decoding with Adapters

Junliang Guo, Zhirui Zhang+4 more|arXiv (Cornell University)|Oct 13, 2020

Topic ModelingComputer Science38 references40 citations

TL;DR

The paper presents AB-Net, a framework that inserts lightweight adapters into two BERT models (source and target) to enable parallel sequence decoding with Mask-Predict, achieving strong NMT performance with half the decoding latency and efficient parameter usage.

ABSTRACT

While large scale pre-trained language models such as BERT have achieved great success on various natural language understanding tasks, how to efficiently and effectively incorporate them into sequence-to-sequence models and the corresponding text generation tasks remains a non-trivial problem. In this paper, we propose to address this problem by taking two different BERT models as the encoder and decoder respectively, and fine-tuning them by introducing simple and lightweight adapter modules, which are inserted between BERT layers and tuned on the task-specific dataset. In this way, we obtain a flexible and efficient model which is able to jointly leverage the information contained in the source-side and target-side BERT models, while bypassing the catastrophic forgetting problem. Each component in the framework can be considered as a plug-in unit, making the framework flexible and task agnostic. Our framework is based on a parallel sequence decoding algorithm named Mask-Predict considering the bi-directional and conditional independent nature of BERT, and can be adapted to traditional autoregressive decoding easily. We conduct extensive experiments on neural machine translation tasks where the proposed method consistently outperforms autoregressive baselines while reducing the inference latency by half, and achieves $36.49$/$33.57$ BLEU scores on IWSLT14 German-English/WMT14 German-English translation. When adapted to autoregressive decoding, the proposed method achieves $30.60$/$43.56$ BLEU scores on WMT14 English-German/English-French translation, on par with the state-of-the-art baseline models.

Motivation & Objective

• Explore leveraging two pre-trained BERT models as encoder and decoder in a seq2seq framework using lightweight adapters.
• Mitigate catastrophic forgetting by freezing BERT parameters and training adapters only.
• Apply a parallel decoding scheme (Mask-Predict) to exploit BERT's bidirectional context while maintaining conditional generation."
• Demonstrate performance gains over autoregressive baselines across multiple translation tasks and languages.

Proposed method

• Insert adapter modules into every BERT layer for both encoder and decoder sides and fine-tune only adapters.
• Use two BERT models (source-side Xbert and target-side Ybert) as encoder/decoder in a seq2seq setup.
• Train with a conditional masked language modeling objective L(y^m|y^r,x; Aenc, Adec) similar to Equation (3).
• Adopt Mask-Predict parallel decoding to leverage BERT’s bidirectional context and enable fast inference; optionally extend to autoregressive decoding.
• Predict target length via a special [LENGTH] token and perform mask-and-predict decoding with iterative refinement.
• Optionally vary adapter architectures (Aenc, Adec) and layer placement to balance performance and parameter efficiency.

Experimental results

Research questions

• RQ1 Can BERT be jointly utilized as both encoder and decoder in a seq2seq framework using adapters?
• RQ2 Does training only adapter modules while freezing BERT layers mitigate catastrophic forgetting and improve efficiency?
• RQ3 Does parallel decoding with Mask-Predict provide speedups and competitive translation quality compared to autoregressive baselines?
• RQ4 How do adapter scales and architectures impact performance and training efficiency?
• RQ5 Is the framework effective across multiple language pairs and resource settings?

Key findings

• AB-Net achieves 36.49 BLEU on IWSLT14 De-En and 33.57 BLEU on WMT14 De-En with parallel decoding, outperforming Mask-Predict and autoregressive baselines.
• AB-Net reduces decoding latency by about 2x compared to Transformer-Base with similar trainable parameter counts.
• AB-Net with dual-side BERT (encoder and decoder) uses fewer trainable parameters than BERT-Fused NAT while achieving higher BLEU than the baseline.
• The encoder- and decoder-side adapters enable the model to leverage information from both BERT models and model conditional dependencies, improving performance.
• AB-Net-Enc (encoder-only BERT with adapters) also yields strong results, and using adapters on the top layers can retain performance with fewer parameters.
• On low-resource IWSLT14 language pairs, AB-Net consistently outperforms baselines across En-It, It-En, En-Es, Es-En, En-Nl, and Nl-En.