[Paper Review] Multi-layer Representation Fusion for Neural Machine Translation

Qiang Wang, Fuxue Li+4 more|arXiv (Cornell University)|Feb 16, 2020

Natural Language Processing Techniques27 references46 citations

TL;DR

This paper introduces multi-layer representation fusion (MLRF) to densely connect all stacked layers in Transformer-based NMT, enabling predictions to utilize lower-level representations via three fusion functions, yielding BLEU gains and regularization benefits, including a new state-of-the-art on IWSLT German-English.

ABSTRACT

Neural machine translation systems require a number of stacked layers for deep models. But the prediction depends on the sentence representation of the top-most layer with no access to low-level representations. This makes it more difficult to train the model and poses a risk of information loss to prediction. In this paper, we propose a multi-layer representation fusion (MLRF) approach to fusing stacked layers. In particular, we design three fusion functions to learn a better representation from the stack. Experimental results show that our approach yields improvements of 0.92 and 0.56 BLEU points over the strong Transformer baseline on IWSLT German-English and NIST Chinese-English MT tasks respectively. The result is new state-of-the-art in German-English translation.

Motivation & Objective

• Motivate the use of lower-layer representations in neural machine translation to reduce information loss and improve training.
• Propose a densely connected fusion mechanism that accesses all lower-layer representations.
• Develop and compare three fusion functions (avg-pooling, FNN, self-attention with hops) for effective fusion.
• Demonstrate gains over a strong Transformer baseline on German-English and Chinese-English translation tasks.
• Show regularization effects and analyze the impact of embeddings and attention hops on performance.

Proposed method

• Introduce a fusion layer that connects to all stacked layers and learns a fused representation φ(Zj) for each target position j.
• Reformulate the output probability to depend on φ(Zj) instead of only the top layer: Pr(yj|y<j,x)=Softmax(Wo·φ(Zj)+bo).
• Compare three fusion strategies: (i) avg-pooling of {zl j}, (ii) concatenation followed by a feed-forward network (FNN) to produce a d-dimensional fused vector, (iii) multi-hop self-attention over the augmented layer representations with layer embeddings to form a 2D representation.
• Explore self-attention with hops, including a layer-embedding El added to each layer representation ˜zl= zl+El, and compute attention energies with a two-layer FNN to obtain hop-wise weights.
• Allow fusion on encoder-only, decoder-only, or both sides; apply layer normalization after the fusion layer for stable training.
• Demonstrate the approach on Transformer-based NMT with 3-layer/6-layer setups (German-English and Chinese-English) and report BLEU improvements.

Experimental results

Research questions

• RQ1 Does densely fusing all lower-layer representations improve translation quality over the top-layer-only prediction in Transformer-based NMT?
• RQ2 Which fusion function (avg-pooling, FNN, self-attention) yields better translation performance, and does fusion on the encoder, decoder, or both sides matter?
• RQ3 How do the number of hops in the self-attention-based fusion and the inclusion of word and layer embeddings affect performance and training stability?
• RQ4 Does MLRF act as a regularizer and reduce overfitting in deep NMT models?
• RQ5 What are the qualitative interpretations of layer attention when using multi-hop fusion?

Key findings

• MLRF improves Transformer baselines, achieving 0.92 BLEU points gain on IWSLT German-English over the strong Transformer baseline.
• MLRF yields 0.56 BLEU points improvement on NIST Chinese-English translation over the baseline.
• Decoder-side fusion generally outperforms encoder-side fusion, and fusion on both sides can offer additional gains depending on the fusion function.
• The best configuration reported is combining FNN-based fusion on the encoder side with self-attention-based fusion on the decoder side (Both-FNN-SA), outperforming the 3-layer baseline by 0.92 BLEU on German-English.
• A self-attention, multi-hop fusion (Dec-SA) provides consistent gains and shows that increasing hops up to a point (nhop < 6) improves performance, with diminishing returns beyond that.
• The approach exhibits a regularizing effect, reducing overfitting compared to the baseline, as shown by training/validation dynamics and dropout analyses.
• Layer attention visualizations indicate hops focus on high-level layers for most predictions, with later hops spreading attention across layers, reflecting different aspects captured by each hop.