[Paper Review] Translating the Unseen? Yoruba-English MT in Low-Resource, Morphologically-Unmarked Settings

Ife Adebara, Muhammad Abdul-Mageed+1 more|arXiv (Cornell University)|Mar 7, 2021

Natural Language Processing Techniques43 references4 citations

TL;DR

This paper evaluates sequence-to-sequence neural machine translation models—specifically SMT, BiLSTM, and Transformer—on Yorùbá-to-English translation in a low-resource setting, focusing on the disambiguation of morphologically unmarked bare nouns (BNs). The Transformer model outperforms both SMT and BiLSTM in correctly disambiguating BNs into definite/indefinite singular/plural categories, achieving 67.85% accuracy, highlighting its superior linguistic generalization in morphologically asymmetric translation tasks.

ABSTRACT

Translating between languages where certain features are marked morphologically in one but absent or marked contextually in the other is an important test case for machine translation. When translating into English which marks (in)definiteness morphologically, from Yorùbá which uses bare nouns but marks these features contextually, ambiguities arise. In this work, we perform fine-grained analysis on how an SMT system compares with two NMT systems (BiLSTM and Transformer) when translating bare nouns in Yorùbá into English. We investigate how the systems what extent they identify BNs, correctly translate them, and compare with human translation patterns. We also analyze the type of errors each model makes and provide a linguistic description of these errors. We glean insights for evaluating model performance in low-resource settings. In translating bare nouns, our results show the transformer model outperforms the SMT and BiLSTM models for 4 categories, the BiLSTM outperforms the SMT model for 3 categories while the SMT outperforms the NMT models for 1 category.

Motivation & Objective

• To evaluate the performance of statistical and neural machine translation models in translating Yorùbá bare nouns (BNs) into English, a language with overt morphological marking of definiteness and number.
• To investigate whether NMT models, particularly Transformers, outperform SMT in low-resource, morphologically unmarked translation settings involving cross-linguistic asymmetries.
• To analyze error types in MT outputs, especially regarding determiner usage, word order, and grammaticality, to assess linguistic generalization beyond BLEU scores.
• To develop a new parallel dataset for Yorùbá–English translation and use it to conduct fine-grained, human-annotated evaluation of BN disambiguation.
• To provide insights into model behavior in low-resource, typologically distant language pairs, with implications for evaluating MT beyond automatic metrics.

Proposed method

• Constructed a new parallel dataset of 1,000 sentence pairs from Yorùbá to English, focusing on contexts where bare nouns require disambiguation into definite/indefinite singular/plural forms.
• Trained three MT systems: a phrase-based SMT system, a sequence-to-sequence BiLSTM model, and a Transformer model, all on the same low-resource Yorùbá–English parallel data.
• Performed human evaluation on test set translations, classifying each BN translation into one of four categories: definite singular, indefinite singular, definite plural, or indefinite plural.
• Defined and applied four error categories: missing words, wrong/misspelled words, grammaticality issues (syntactic/semantic), and incorrect word order (including phrase-level reordering).
• Used gold-standard human translations as reference to compute disambiguation accuracy and compare model outputs across categories.
• Correlated disambiguation accuracy with BLEU scores and analyzed frequency effects, finding higher accuracy for more frequent BN categories.

Experimental results

Research questions

• RQ1 How do SMT, BiLSTM, and Transformer models perform in disambiguating Yorùbá bare nouns into English definite/indefinite singular/plural forms?
• RQ2 Does the Transformer model outperform SMT and BiLSTM in low-resource, morphologically asymmetric translation, particularly for BN disambiguation?
• RQ3 What types of errors do these models make when translating bare nouns, and how do error patterns relate to linguistic phenomena like word order, determiner usage, and grammaticality?
• RQ4 Is there a correlation between disambiguation accuracy and BLEU scores, and does model performance vary by BN category frequency?
• RQ5 To what extent do model outputs reflect contextual cues (e.g., familiarity, uniqueness, verb category) for BN disambiguation?

Key findings

• The Transformer model achieved the highest BN disambiguation accuracy at 67.85%, outperforming the BiLSTM (63.83%) and SMT (59.12%) models.
• The Transformer outperformed both SMT and BiLSTM in four of the five BN disambiguation categories, while BiLSTM outperformed SMT in three categories and SMT outperformed NMT in one category.
• All three models showed the highest accuracy in translating Yorùbá bare nouns into definite singular English forms, suggesting a bias toward more frequent or semantically salient categories.
• A positive correlation was found between disambiguation accuracy and BLEU scores, indicating that higher BLEU scores were generally associated with better disambiguation performance.
• Error analysis revealed that even when BNs were correctly categorized, models frequently made grammaticality errors (e.g., incorrect tense, punctuation, or word order), and 12.5% of errors were due to missing or incorrect determiners.
• The study demonstrates that BLEU scores alone are insufficient for evaluating linguistic generalization in low-resource MT, especially for morphosyntactic asymmetries like those in Yorùbá–English translation.