[Paper Review] Neural Machine Translation with Joint Representation

Yanyang Li, Qiang Wang+3 more|arXiv (Cornell University)|Feb 16, 2020

Natural Language Processing Techniques12 references4 citations

TL;DR

This paper proposes Reformer, a novel sequence-to-sequence framework for neural machine translation that leverages joint representation across source and target sequences via a new efficient attention mechanism called Separable Attention. By fully modeling interactions between all source and target tokens while maintaining computational efficiency, Reformer outperforms the Transformer baseline by up to 1.9 BLEU points and achieves state-of-the-art results with 50% fewer parameters on large-scale translation tasks.

ABSTRACT

Though early successes of Statistical Machine Translation (SMT) systems are attributed in part to the explicit modelling of the interaction between any two source and target units, e.g., alignment, the recent Neural Machine Translation (NMT) systems resort to the attention which partially encodes the interaction for efficiency. In this paper, we employ Joint Representation that fully accounts for each possible interaction. We sidestep the inefficiency issue by refining representations with the proposed efficient attention operation. The resulting Reformer models offer a new Sequence-to- Sequence modelling paradigm besides the Encoder-Decoder framework and outperform the Transformer baseline in either the small scale IWSLT14 German-English, English-German and IWSLT15 Vietnamese-English or the large scale NIST12 Chinese-English translation tasks by about 1 BLEU point.We also propose a systematic model scaling approach, allowing the Reformer model to beat the state-of-the-art Transformer in IWSLT14 German-English and NIST12 Chinese-English with about 50% fewer parameters. The code is publicly available at https://github.com/lyy1994/reformer.

Motivation & Objective

• To address the inefficiency of full joint representation in neural machine translation, which explicitly models all source-target unit interactions but incurs high computational cost.
• To develop an efficient attention mechanism that fully captures interactions between source and target sequences, overcoming the limitations of standard attention that restricts interaction to single sequences.
• To propose a new sequence-to-sequence modeling paradigm based on joint representation, independent of the traditional encoder-decoder framework.
• To enable systematic model scaling that achieves SOTA performance with significantly reduced parameter count.
• To improve translation quality—especially for long sequences and rare tokens—by enhancing context modeling and length prediction accuracy.

Proposed method

• Introduces Separable Attention, a novel attention mechanism that operates over the joint representation space $\mathbb{R}^{S\times T\times H}$, enabling full interaction modeling between source and target tokens.
• Designs two Reformer variants: Reformer-base for maximum effectiveness with $O(1)$ access to any source or target token, and Reformer-fast for improved efficiency with comparable performance.
• Employs a reformulated self-attention mechanism that splits computation across source and target sequences, reducing complexity while preserving full interaction modeling.
• Applies residual connections and layer normalization to stabilize training, following the Transformer architecture but adapted to the joint representation space.
• Proposes a systematic model scaling approach requiring only $O(2)$ runs to generate enhanced models, in contrast to polynomial-cost grid search.
• Utilizes a shared parameter initialization strategy across scaled models to ensure consistent performance gains without exhaustive hyperparameter tuning.

Experimental results

Research questions

• RQ1 Can a sequence-to-sequence model based on full joint representation outperform standard Transformer models in translation quality while remaining computationally efficient?
• RQ2 How does Separable Attention compare to standard self-attention in modeling long-range dependencies and rare token prediction?
• RQ3 To what extent can model scaling improve performance with fewer parameters, and is this scalable approach more efficient than grid search?
• RQ4 Does joint representation lead to better length prediction and reduced under-translation compared to standard attention mechanisms?
• RQ5 Can the Reformer framework generalize across both small-scale and large-scale translation benchmarks?

Key findings

• Reformer-base and Reformer-fast outperform the Transformer baseline by 1.3, 0.8, and 0.7 BLEU points on IWSLT14 German-English, English-German, and IWSLT15 Vietnamese-English, respectively.
• On the large-scale NIST12 Chinese-English benchmark, Reformer achieves a 1.9 BLEU point improvement over the Transformer baseline.
• After systematic model scaling, Reformer beats the SOTA large Transformer by 0.7 BLEU on IWSLT14 German-English and 2.0 BLEU on NIST12 Chinese-English with approximately 50% fewer parameters.
• Reformer models generate translations with more appropriate lengths, showing a higher length ratio than the baseline and reducing under-translation.
• Reformer achieves significantly higher prediction accuracy for distant target positions and low-frequency tokens, indicating better context modeling and long-range dependency capture.
• Visualizations of separable attention show dynamic, token-specific attention distributions across both source and target sequences, unlike the static distributions in standard self-attention.