[Paper Review] Application of Low-resource Machine Translation Techniques to Russian-Tatar Language Pair

Aidar Valeev, Ilshat Gibadullin+2 more|arXiv (Cornell University)|Oct 1, 2019

Natural Language Processing Techniques13 references7 citations

TL;DR

This paper applies low-resource neural machine translation techniques—specifically transfer learning, back-translation, and Shallow Fusion—to the underexplored Russian-Tatar language pair with a small parallel corpus of 324,000 sentences. Using a Transformer-based model, transfer learning from a related Kazakh-Russian pair yielded the highest gains, improving translation quality by +2.89 BLEU (Tatar-to-Russian) and +2.57 BLEU (Russian-to-Tatar) over baseline.

ABSTRACT

Neural machine translation is the current state-of-the-art in machine translation. Although it is successful in a resource-rich setting, its applicability for low-resource language pairs is still debatable. In this paper, we explore the effect of different techniques to improve machine translation quality when a parallel corpus is as small as 324 000 sentences, taking as an example previously unexplored Russian-Tatar language pair. We apply such techniques as transfer learning and semi-supervised learning to the base Transformer model, and empirically show that the resulting models improve Russian to Tatar and Tatar to Russian translation quality by +2.57 and +3.66 BLEU, respectively.

Motivation & Objective

• Address the lack of research on low-resource neural machine translation between Slavic and Turkic languages, particularly Russian-Tatar.
• Improve translation quality for the Russian-Tatar language pair despite limited parallel monolingual and parallel corpora.
• Evaluate the effectiveness of transfer learning, back-translation, and language model integration in a low-resource setting.
• Investigate the interaction between model quality and back-translation efficacy, especially in relation to initialization quality.

Proposed method

• Fine-tune a Transformer-based NMT model on a small parallel corpus of 324,000 Russian-Tatar sentence pairs.
• Apply transfer learning by initializing the model with pre-trained weights from a related Kazakh-Russian language pair, training for 1 or 6 epochs.
• Use back-translation by generating synthetic parallel data via monolingual Tatar and Russian corpora using the initial NMT model.
• Integrate a pre-trained language model into the NMT model using Shallow Fusion, combining NMT and LM logits with a learned weighting hyperparameter.
• Train the final model on the combined real and synthetic parallel data, with careful validation to avoid overfitting.
• Optimize hyperparameters such as the Shallow Fusion weight and synthetic-to-real data ratio to improve generalization.

Experimental results

Research questions

• RQ1 How effective is transfer learning from a related low-resource language pair (Kazakh-Russian) in improving Russian-Tatar translation quality?
• RQ2 Does back-translation consistently improve performance across different model initialization stages and translation directions?
• RQ3 Can Shallow Fusion with a pre-trained language model significantly enhance translation quality in a low-resource setting?
• RQ4 How does the quality of the initial model affect the effectiveness of back-translation?
• RQ5 What is the optimal combination of techniques (transfer learning, back-translation, LM integration) for low-resource Slavic-Turkic translation?

Key findings

• Transfer learning from a 1-epoch Kazakh-Russian initialization improved Russian-to-Tatar translation by +2.57 BLEU (case-sensitive) and Tatar-to-Russian by +2.89 BLEU (case-insensitive).
• Extending transfer learning to 6 epochs further improved results, yielding +2.89 BLEU (TT-RU) and +2.57 BLEU (RU-TT) over baseline.
• Back-translation provided significant gains only when applied to lower-quality models; when applied to 6-epoch initialized models, it yielded negligible or even negative improvements.
• Shallow Fusion integration of a language model failed to improve performance, with only +0.01 BLEU gain under optimal hyperparameter settings.
• The effectiveness of back-translation was inversely related to the quality of the underlying model, suggesting diminishing returns with better initialization.
• The best overall result was achieved with back-translation applied to a 1-epoch Kazakh-initialized model in the Tatar-to-Russian direction, reaching 30.06 BLEU (case-insensitive).