[Paper Review] How do lexical semantics affect translation? An empirical study

Vivek Subramanian, Dhanasekar Sundararaman|arXiv (Cornell University)|Dec 31, 2021

Natural Language Processing Techniques4 citations

TL;DR

This paper investigates how lexical semantics—specifically word order and lexical similarity—affect neural machine translation (NMT) performance. Using English-to-15 low-resource language pairs from OpenSubtitles2016, it shows that translation performance increases with lexical similarity to English and that POS supervision significantly boosts performance for languages with dissimilar word orders, especially in Transformer models.

ABSTRACT

Neural machine translation (NMT) systems aim to map text from one language into another. While there are a wide variety of applications of NMT, one of the most important is translation of natural language. A distinguishing factor of natural language is that words are typically ordered according to the rules of the grammar of a given language. Although many advances have been made in developing NMT systems for translating natural language, little research has been done on understanding how the word ordering of and lexical similarity between the source and target language affect translation performance. Here, we investigate these relationships on a variety of low-resource language pairs from the OpenSubtitles2016 database, where the source language is English, and find that the more similar the target language is to English, the greater the translation performance. In addition, we study the impact of providing NMT models with part of speech of words (POS) in the English sequence and find that, for Transformer-based models, the more dissimilar the target language is from English, the greater the benefit provided by POS.

Motivation & Objective

• To understand how lexical semantics, particularly word order and lexical similarity, influence neural machine translation (NMT) performance.
• To evaluate the impact of part-of-speech (POS) tagging as explicit linguistic supervision on source-side tokens in NMT models.
• To compare LSTM and Transformer architectures in handling translation across diverse linguistic typologies.
• To assess whether linguistic similarity between source and target languages affects translation quality in low-resource settings.
• To determine whether POS supervision provides greater benefit for languages with dissimilar syntactic structures to English.

Proposed method

• Empirical study using 15 low-resource language pairs from the OpenSubtitles2016 dataset, with English as the source language.
• Training both LSTM-based and Transformer-based NMT models with and without explicit part-of-speech (POS) tagging on source-side tokens.
• Applying byte pair encoding (BPE) with a shared subword vocabulary for both source and target languages.
• Appending trainable POS embedding vectors to subword token embeddings, following Sennrich and Haddow (2016) and Sundararaman et al. (2019).
• Evaluating translation quality using BLEU scores on held-out test sets for each language pair.
• Classifying target languages by dominant word order (SVO, SOV, or flexible) and computing normalized Levenshtein distance to English for lexical similarity.

Experimental results

Research questions

• RQ1 How does the syntactic word order of the target language (e.g., SVO vs. SOV) affect translation performance in NMT models?
• RQ2 To what extent does lexical similarity between English and the target language influence BLEU scores in low-resource NMT?
• RQ3 Does providing part-of-speech (POS) information to the source-side decoder improve translation performance, particularly for languages with dissimilar word order to English?
• RQ4 How do LSTM and Transformer architectures differ in their sensitivity to linguistic differences such as word order and lexical similarity?
• RQ5 Is the benefit of POS supervision more pronounced in models trained on low-resource, typologically distant language pairs?

Key findings

• Translation performance, measured by BLEU score, increases with higher lexical similarity between English and the target language, as indicated by lower normalized Levenshtein distance.
• For Transformer-based models, the benefit of POS supervision is most significant for languages with word orders most dissimilar to English’s SVO structure, such as Sinhala (SOV) and Basque (flexible/SOV).
• The LSTM POS model achieves a BLEU score of 22.26 on Hindi (SOV), compared to 22.26 for the baseline LSTM, indicating minimal improvement, while the Transformer POS model improves from 23.80 to 24.15 on the same language.
• The Transformer POS model achieves a BLEU score of 29.23 on Catalan (SVO), a 1.93-point improvement over the baseline Transformer (27.30), indicating strong gains for languages with similar word order.
• For highly dissimilar languages like Basque (Levenshtein distance 0.407), the Transformer POS model improves BLEU from 16.60 to 17.80, showing a 1.2-point gain, suggesting POS helps bridge syntactic gaps.
• The study confirms that linguistic features such as word order and lexical similarity are critical factors in NMT performance, especially in low-resource settings, and that POS supervision can mitigate performance drops in structurally distant language pairs.