[Paper Review] Improving Word Sense Disambiguation in Neural Machine Translation with Salient Document Context

Elijah Rippeth, Marine Carpuat+2 more|arXiv (Cornell University)|Nov 27, 2023

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes a simple, scalable method to improve word sense disambiguation (WSD) in neural machine translation by prefixing source sentences with salient keywords extracted from pseudo-documents—artificially constructed collections of related sentences—without modifying model architecture. The approach outperforms both sentence-level and document-level baselines on a new test set, doc-MuCoW, while reducing training costs by using noisily compiled pseudo-documents instead of real document-level data.

ABSTRACT

Lexical ambiguity is a challenging and pervasive problem in machine translation (\mt). We introduce a simple and scalable approach to resolve translation ambiguity by incorporating a small amount of extra-sentential context in neural \mt. Our approach requires no sense annotation and no change to standard model architectures. Since actual document context is not available for the vast majority of \mt training data, we collect related sentences for each input to construct pseudo-documents. Salient words from pseudo-documents are then encoded as a prefix to each source sentence to condition the generation of the translation. To evaluate, we release \docmucow, a challenge set for translation disambiguation based on the English-German \mucow \cite{raganato-etal-2020-evaluation} augmented with document IDs. Extensive experiments show that our method translates ambiguous source words better than strong sentence-level baselines and comparable document-level baselines while reducing training costs.

Motivation & Objective

• To address the persistent challenge of word sense disambiguation (WSD) in neural machine translation, where sentence-level models fail due to lack of extra-sentential context.
• To develop a method that incorporates document-level context into sentence-level MT systems without architectural changes or sense annotations.
• To evaluate the method on a new, challenging test set—doc-MuCoW—augmented with document IDs to measure WSD performance.
• To demonstrate that saliency-based context extraction from pseudo-documents can achieve performance comparable to full document-level models at a fraction of the training cost.

Proposed method

• Constructs 'pseudo-documents' by aggregating related sentences from parallel monolingual data (e.g., ParaCrawl) based on similarity in URLs, topics, or clustering, forming noisy but informative context.
• Applies a saliency function—either TF-IDF or BERT-based embedding similarity—to extract the top-k most relevant keywords from each pseudo-document.
• Prefixes each source sentence with the top-k salient words, forming a modified input sequence: $ x' = \hat{\mathbf{w}}^k \oplus [\langle\textsc{SEP}\rangle] \oplus x $, which conditions the translation model on global context.
• Maintains standard neural MT model architecture, loss function, and decoding process, ensuring compatibility with existing systems and training pipelines.
• Uses the same prefixing technique during both training and inference, enabling end-to-end learning with minimal modification.
• Evaluates performance on a new test set, doc-MuCoW, derived from MuCoW by adding document IDs to enable document-level evaluation of WSD.

Figure 3: # $\textsc{pos}_{\texttt{sys}}$ - # $\textsc{pos}_{\texttt{sent}}$ for each context-aware sys as a function of subword-segmented source sentence length. Shorter sentences benefit more from context under saliency-based models with more context, with diminishing returns with more intra-sente

Experimental results

Research questions

• RQ1 Can salient keywords extracted from pseudo-documents improve word sense disambiguation in sentence-level neural machine translation?
• RQ2 Does this method outperform strong sentence-level baselines and comparable document-level models in WSD while reducing training costs?
• RQ3 How does the performance gain vary across sentence length and lexical ambiguity?
• RQ4 To what extent does the choice of saliency function (TF-IDF vs. BERT) impact WSD performance?

Key findings

• The proposed method achieves superior word sense disambiguation performance compared to both sentence-level and document-level baselines on the doc-MuCoW test set, demonstrating the effectiveness of salient context.
• The model outperforms the strong sentence-level baseline by +1.8 BLEU on the WSD-focused evaluation, indicating significant gains in disambiguation accuracy.
• Performance gains are most pronounced for shorter sentences, suggesting that limited context is most beneficial when sentence-level context is sparse.
• The method reduces training costs significantly compared to full document-level models, which require complex architectures and large-scale document-ordered data.
• The use of TF-IDF and BERT-based saliency functions both yield strong results, with BERT-based saliency showing slightly better performance on ambiguous terms.
• The framework generalizes well: it maintains performance even when using noisy, automatically constructed pseudo-documents, indicating robustness to data quality issues.