[Paper Review] Deep Neural Machine Translation with Linear Associative Unit

Mingxuan Wang, Zhengdong Lu+2 more|arXiv (Cornell University)|May 2, 2017

Natural Language Processing Techniques24 references14 citations

TL;DR

This paper proposes the Linear Associative Unit (LAU), a novel recurrent neural network unit that enables unimpeded gradient flow through both time and space by fusing linear and nonlinear transformations. The resulting DeepLAU model achieves state-of-the-art performance on Chinese-English translation (11.7 BLEU improvement over Groundhog) and competitive results on English-German and English-French WMT14 tasks, demonstrating superior training stability and performance in deep architectures.

ABSTRACT

Deep Neural Networks (DNNs) have provably enhanced the state-of-the-art Neural Machine Translation (NMT) with their capability in modeling complex functions and capturing complex linguistic structures. However NMT systems with deep architecture in their encoder or decoder RNNs often suffer from severe gradient diffusion due to the non-linear recurrent activations, which often make the optimization much more difficult. To address this problem we propose novel linear associative units (LAU) to reduce the gradient propagation length inside the recurrent unit. Different from conventional approaches (LSTM unit and GRU), LAUs utilizes linear associative connections between input and output of the recurrent unit, which allows unimpeded information flow through both space and time direction. The model is quite simple, but it is surprisingly effective. Our empirical study on Chinese-English translation shows that our model with proper configuration can improve by 11.7 BLEU upon Groundhog and the best reported results in the same setting. On WMT14 English-German task and a larger WMT14 English-French task, our model achieves comparable results with the state-of-the-art.

Motivation & Objective

• To address the vanishing gradient problem in deep recurrent neural networks used for neural machine translation.
• To reduce gradient path length within recurrent units to improve optimization in deep encoder-decoder architectures.
• To design a simple yet effective recurrent unit that supports deeper networks without performance degradation.
• To achieve state-of-the-art or competitive results on major NMT benchmarks using deeper models.
• To demonstrate that depth, when combined with efficient gradient flow, is more beneficial than width for modeling complex linguistic structures.

Proposed method

• Proposes the Linear Associative Unit (LAU), which combines linear and nonlinear transformations within a recurrent unit to enable direct information flow.
• The LAU computes a linear sum between input and hidden state, while also applying a nonlinear transformation, allowing both short-cut and sequential paths.
• The architecture enables unimpeded gradient propagation across time and space, reducing vanishing gradient effects.
• Employs a deep encoder and decoder with LAUs, allowing for deeper stacking without optimization collapse.
• Uses standard attention mechanisms and sequence-to-sequence training with cross-entropy loss.
• Evaluates model performance using BLEU scores on standard NMT benchmarks, including NIST Chinese-English and WMT14 English-German/French.

Experimental results

Research questions

• RQ1 Can a novel recurrent unit design reduce gradient path length and improve training stability in deep NMT models?
• RQ2 Does enabling direct linear information flow through recurrent units lead to better performance on long and complex sentences?
• RQ3 Can deeper networks with LAUs outperform shallower models even with fewer parameters?
• RQ4 How does the LAU compare to residual and fast-forward connections in terms of BLEU score and training stability?
• RQ5 To what extent does model depth, rather than width, contribute to performance gains in neural machine translation?

Key findings

• On the NIST Chinese-English translation task, DeepLAU achieved a 11.7 BLEU improvement over Groundhog and a 4.9 BLEU gain over a strong NMT baseline with standard techniques.
• With 8-layer encoder and 6-layer decoder, DeepLAU achieved a BLEU score of 39.01, outperforming DeepGRU and other baselines.
• Even with fewer parameters, DeepLAU with depth (8,6) outperformed wider DeepGRU models, indicating depth is more beneficial than width.
• On WMT14 English-German and English-French tasks, DeepLAU achieved results comparable to state-of-the-art models.
• DeepLAU maintained performance gains on longer sentences, with consistently higher BLEU scores than DeepGRU across all length bins.
• The model showed stable training even at extreme depths (e.g., 8 layers in encoder), where standard DeepGRU models degraded in performance.