[Paper Review] Dense Information Flow for Neural Machine Translation

Yanyao Shen, Xu Tan+3 more|arXiv (Cornell University)|Jun 3, 2018

Multimodal Machine Learning Applications13 references4 citations

TL;DR

This paper proposes DenseNMT, a neural machine translation architecture that uses dense connections in both encoder and decoder blocks and a dense attention mechanism to improve information flow, resulting in faster training and higher translation accuracy. Experiments show it achieves new SOTA BLEU scores on IWSLT14 and WMT14 benchmarks with fewer parameters and reduced embedding size.

ABSTRACT

Recently, neural machine translation has achieved remarkable progress by introducing well-designed deep neural networks into its encoder-decoder framework. From the optimization perspective, residual connections are adopted to improve learning performance for both encoder and decoder in most of these deep architectures, and advanced attention connections are applied as well. Inspired by the success of the DenseNet model in computer vision problems, in this paper, we propose a densely connected NMT architecture (DenseNMT) that is able to train more efficiently for NMT. The proposed DenseNMT not only allows dense connection in creating new features for both encoder and decoder, but also uses the dense attention structure to improve attention quality. Our experiments on multiple datasets show that DenseNMT structure is more competitive and efficient.

Motivation & Objective

• To improve information flow in neural machine translation by replacing residual connections with dense connections in encoder and decoder networks.
• To enhance attention quality by introducing a dense attention structure that allows all previous encoder layers to directly influence the decoder.
• To enable more efficient training and better performance with reduced model size, particularly using smaller embedding dimensions.
• To achieve state-of-the-art translation performance on multiple benchmark datasets without increasing model parameters.

Proposed method

• Introduces a densely connected encoder and decoder architecture where each layer concatenates features from all previous layers, promoting feature reuse and improved gradient flow.
• Applies dense connections not only within the encoder and decoder but also across the attention mechanism, enabling all encoder layers to contribute directly to attention computation.
• Uses a modified attention mechanism that aggregates representations from all encoder layers, improving alignment quality and semantic representation.
• Employs a standard encoder-decoder framework with convolutional or self-attention-based transformations, but replaces residual skip connections with dense skip connections.
• Trains models using standard NMT objectives (e.g., cross-entropy loss) with Adam optimization, and evaluates using BLEU score on standard benchmarks.
• Conducts ablation studies to isolate the contribution of dense connections in the encoder, decoder, and attention components.

Experimental results

Research questions

• RQ1 Can dense connections in both encoder and decoder improve training efficiency and translation performance in NMT compared to residual connections?
• RQ2 Does a dense attention mechanism that aggregates features from all encoder layers lead to better alignment and higher-quality translations?
• RQ3 Can DenseNMT achieve comparable or better performance with significantly smaller embedding sizes compared to standard models?
• RQ4 How does DenseNMT compare to state-of-the-art models in terms of BLEU score, parameter count, and training efficiency on major NMT benchmarks?

Key findings

• DenseNMT achieves a new SOTA BLEU score of 32.26 on the IWSLT14 German-English translation task, outperforming prior models by up to 2.8 BLEU points.
• On the WMT14 English-German task, DenseNMT achieves 25.52 BLEU with 20% fewer parameters and 35% fewer training iterations than the ConvS2S baseline.
• The 8-layer DenseNMT model with embedding size 64 matches the performance of a standard 8-layer model with embedding size 256, while using only 40% of the parameters.
• Ablation studies confirm that dense connections in all three components (encoder, decoder, attention) contribute synergistically to performance without increasing model size.
• DenseNMT achieves a 24.36 average BLEU score on the Turkish-English IWSLT14 task, setting a new benchmark.
• The model shows faster convergence, as evidenced by steeper and earlier-declining training loss curves, indicating more efficient optimization.