[Paper Review] Augmenting Statistical Machine Translation with Subword Translation of Out-of-Vocabulary Words

Nelson F. Liu, Jonathan May+2 more|arXiv (Cornell University)|Aug 16, 2018

Natural Language Processing Techniques39 references9 citations

TL;DR

This paper proposes subword-based methods to improve statistical machine translation (SMT) by translating out-of-vocabulary (OOV) words using orthographic clues. It introduces three generalizable approaches—edit distance, vector distance, and character-level sequence-to-sequence models—demonstrating consistent BLEU gains (0.5 average, up to 2.0) across 14 typologically diverse languages, especially in low-resource settings.

ABSTRACT

Most statistical machine translation systems cannot translate words that are unseen in the training data. However, humans can translate many classes of out-of-vocabulary (OOV) words (e.g., novel morphological variants, misspellings, and compounds) without context by using orthographic clues. Following this observation, we describe and evaluate several general methods for OOV translation that use only subword information. We pose the OOV translation problem as a standalone task and intrinsically evaluate our approaches on fourteen typologically diverse languages across varying resource levels. Adding OOV translators to a statistical machine translation system yields consistent BLEU gains (0.5 points on average, and up to 2.0) for all fourteen languages, especially in low-resource scenarios.

Motivation & Objective

• Address the persistent challenge in statistical machine translation (SMT) of translating out-of-vocabulary (OOV) words unseen during training.
• Overcome SMT's limitation in handling OOVs by leveraging subword orthographic information, inspired by human ability to infer translations from morphological and orthographic patterns.
• Develop general, language-agnostic OOV translation methods that do not rely on language-specific heuristics or extensive parallel data.
• Evaluate the intrinsic performance of OOV translation methods across diverse languages and assess their impact when integrated into a full SMT system.
• Demonstrate that subword-level reasoning enables the generation of novel target-side words not present in training data, especially through sequence-to-sequence models.

Proposed method

• Use edit distance to match OOV tokens to in-vocabulary foreign words with minimal Levenshtein distance, then predict the most frequent English translation of the matched words.
• Apply FastText word embeddings trained on monolingual data to compute cosine similarity between OOV and in-vocabulary word vectors, selecting the closest match for translation prediction.
• Train character-level sequence-to-sequence (seq2seq) models to generate English translations token-by-token from OOV input, enabling composition of novel target-side words.
• Construct OOV translators using monolingual data, bilingual lexicons, and word-aligned translation tables extracted from parallel corpora, without requiring language-specific rules.
• Evaluate methods intrinsically on 14 language pairs using held-out OOV word pairs not present in training data, with separate validation and test sets.
• Integrate the best-performing OOV translator into a syntax-based SMT (SBMT) system and measure overall BLEU improvement on standard test sets.

Experimental results

Research questions

• RQ1 Can subword-level orthographic similarity be effectively leveraged to translate OOV words in a way that generalizes across diverse languages?
• RQ2 How do different subword-based OOV translation strategies—edit distance, vector distance, and seq2seq—compare in performance across various OOV types?
• RQ3 To what extent do OOV translators improve overall SMT system performance, especially in low-resource settings?
• RQ4 Can seq2seq models generate novel English words not seen during training by recombining subword units?
• RQ5 Which OOV types (e.g., morphological variants, compounds, misspellings) are best handled by each method, and why?

Key findings

• Adding OOV translators to a syntax-based SMT system yields consistent BLEU improvements across all 14 languages, with an average gain of 0.5 BLEU and up to 2.0 BLEU in low-resource settings.
• The sequence-to-sequence (seq2seq) model achieves the highest performance on morphological variants and compound words, with 24.6% exact-match accuracy on morphological variations and 40.0% on compounds, outperforming edit and vector methods.
• Edit distance performs best on misspellings (31.6% accuracy), while vector distance shows stronger performance on proper nouns (20.0% accuracy), though seq2seq still outperforms it (60.0% accuracy).
• The seq2seq model successfully generates novel English words not present in the training data, such as 'cyberviolence' and 'balkanized', demonstrating its ability to compose subword units into unseen target-side tokens.
• The edit distance method achieves 21.4% accuracy on transliteration cases, matching the seq2seq model, while vector distance lags at 14.3%.
• The intrinsic evaluation shows that the seq2seq model generalizes best across diverse OOV types, with 26.0% overall exact-match accuracy, significantly outperforming edit (17.0%) and vector (14.0%) methods.