[Paper Review] Attention Focusing for Neural Machine Translation by Bridging Source and Target Embeddings

Shaohui Kuang, Junhui Li+3 more|arXiv (Cornell University)|Nov 15, 2017

Natural Language Processing Techniques19 references3 citations

TL;DR

This paper proposes three bridging models—source-side, target-side, and direct bridging—that shorten the distance between source and target word embeddings in neural machine translation (NMT) to improve word alignment and translation quality. By integrating source and target embeddings more closely within the seq2seq framework, the models significantly enhance alignment accuracy and reduce over/under-translation, outperforming standard NMT without external resources.

ABSTRACT

In neural machine translation, a source sequence of words is encoded into a vector from which a target sequence is generated in the decoding phase. Differently from statistical machine translation, the associations between source words and their possible target counterparts are not explicitly stored. Source and target words are at the two ends of a long information processing procedure, mediated by hidden states at both the source encoding and the target decoding phases. This makes it possible that a source word is incorrectly translated into a target word that is not any of its admissible equivalent counterparts in the target language. In this paper, we seek to somewhat shorten the distance between source and target words in that procedure, and thus strengthen their association, by means of a method we term bridging source and target word embeddings. We experiment with three strategies: (1) a source-side bridging model, where source word embeddings are moved one step closer to the output target sequence; (2) a target-side bridging model, which explores the more relevant source word embeddings for the prediction of the target sequence; and (3) a direct bridging model, which directly connects source and target word embeddings seeking to minimize errors in the translation of ones by the others. Experiments and analysis presented in this paper demonstrate that the proposed bridging models are able to significantly improve quality of both sentence translation, in general, and alignment and translation of individual source words with target words, in particular.

Motivation & Objective

• To address poor word alignment in neural machine translation (NMT), where source and target words lose semantic connection across long hidden state sequences.
• To reduce incorrect alignments—such as misaligning 'eos' with 'late' or 'winter olympics' with a comma—common in NMT but rare in SMT.
• To strengthen the association between source and target word embeddings without relying on external resources like SMT word alignments or lexical tables.
• To improve translation quality by enabling the model to learn more plausible translation pairs directly from training data.

Proposed method

• Source-side bridging: Concatenates each source word embedding with its corresponding source hidden state to improve attention focus on relevant source words.
• Target-side bridging: Selects high-attention source words based on attention weights and integrates them more closely with target hidden states during decoding.
• Direct bridging: Learns a linear transformation matrix to map source embeddings directly to target embeddings, minimizing translation errors.
• The bridging mechanisms are integrated into the standard seq2seq NMT architecture with minimal architectural changes, preserving end-to-end training.
• All models are trained end-to-end using standard NMT objectives, with no additional supervision or pre-trained alignment data.
• The direct bridging model uses a learned transformation matrix to find the closest target word embeddings for each source word, mimicking SMT lexical tables.

Experimental results

Research questions

• RQ1 Can shortening the distance between source and target word embeddings in the NMT processing chain improve word alignment quality?
• RQ2 Does bridging source and target embeddings reduce over-translation and under-translation issues in NMT?
• RQ3 Can the model learn more accurate translation pairs without relying on external SMT word alignments or lexical dictionaries?
• RQ4 How do different bridging strategies (source-side, target-side, direct) compare in improving alignment and translation quality?

Key findings

• The proposed bridging models significantly improve translation quality on Chinese-to-English translation, outperforming the standard NMT baseline.
• The direct bridging model produces translation pairs that are highly consistent with those from SMT lexical tables, indicating strong alignment with admissible translation equivalents.
• The models reduce the incidence of incorrect alignments, such as misaligning 'eos' with 'late' or 'winter olympics' with a comma, which occur in up to 50% of cases in the baseline NMT.
• Experiments show that the bridging mechanisms improve word alignment quality and alleviate over- and under-translation problems.
• The direct bridging model learns a transformation matrix that effectively maps source embeddings to their most likely target counterparts, demonstrating strong capability in capturing translation equivalence.
• Analysis confirms that the bridging methods help the model learn more favorable source-target word pairs, enhancing both global translation quality and local alignment accuracy.