[Paper Review] Triangular Architecture for Rare Language Translation

Shuo Ren, Wenhu Chen+4 more|arXiv (Cornell University)|May 13, 2018

Natural Language Processing Techniques24 references16 citations

TL;DR

This paper proposes a triangular architecture (TA-NMT) that leverages a rich intermediate language Y to improve low-resource neural machine translation (NMT) for rare language pairs (X,Z) by jointly training translation models on (X,Y) and (Y,Z) using a unified bidirectional EM algorithm. The method significantly boosts translation quality on MultiUN and IWSLT2012, especially when combined with back-translation, demonstrating strong gains in low-resource settings.

ABSTRACT

Neural Machine Translation (NMT) performs poor on the low-resource language pair $(X,Z)$, especially when $Z$ is a rare language. By introducing another rich language $Y$, we propose a novel triangular training architecture (TA-NMT) to leverage bilingual data $(Y,Z)$ (may be small) and $(X,Y)$ (can be rich) to improve the translation performance of low-resource pairs. In this triangular architecture, $Z$ is taken as the intermediate latent variable, and translation models of $Z$ are jointly optimized with a unified bidirectional EM algorithm under the goal of maximizing the translation likelihood of $(X,Y)$. Empirical results demonstrate that our method significantly improves the translation quality of rare languages on MultiUN and IWSLT2012 datasets, and achieves even better performance combining back-translation methods.

Motivation & Objective

• To address the poor performance of neural machine translation (NMT) on low-resource language pairs, especially when the target language Z is rare.
• To exploit bilingual data from a rich-resource pair (X,Y) and a low-resource pair (Y,Z) to improve translation quality for the low-resource pair (X,Z).
• To develop a unified training framework that jointly optimizes multiple translation models using an EM-based approach.
• To enable mutual improvement between translation models via pseudo-data generation and reinforcement-like reward signals.

Proposed method

• Introduce a rich intermediate language Y to create a triangular architecture linking source X, target Z, and Y, enabling data and model-level knowledge transfer.
• Model the translation from X to Y via a latent variable Z, decomposing X→Y into two stages: X→Z and Z→Y, both trained jointly.
• Use a unified bidirectional EM algorithm to optimize four translation models: p(z|x), p(x|z), p(z|y), and p(y|z), enabling mutual refinement.
• In the E-step, generate pseudo-bilingual data using current model parameters and assign time-step rewards based on log-likelihood terms, mimicking a reinforcement learning update.
• Leverage monolingual data through back-translation as a complementary technique to further enhance performance.
• Apply the EM framework in both directions: X→Y via Z and Y→X via Z, enabling symmetric model updates.

Experimental results

Research questions

• RQ1 Can introducing a rich intermediate language Y improve translation performance for low-resource language pairs (X,Z)?
• RQ2 How can bilingual data from (X,Y) and (Y,Z) be jointly leveraged to enhance the translation model of (X,Z)?
• RQ3 Can a unified bidirectional EM framework effectively optimize multiple translation models and enable mutual improvement?
• RQ4 Does the proposed method outperform existing low-resource NMT techniques such as back-translation and knowledge distillation?
• RQ5 Can the method be further enhanced by combining with monolingual data augmentation techniques like back-translation?

Key findings

• The TA-NMT method achieves significant improvements in translation quality on the MultiUN and IWSLT2012 datasets for rare language pairs.
• On the IWSLT2012 English-Hebrew translation task, the method outperforms baseline RNNSearch, knowledge distillation, and back-translation methods.
• The joint optimization via the bidirectional EM framework enables mutual improvement between the (X,Z) and (Y,Z) translation models.
• When combined with back-translation, TA-NMT achieves even greater performance gains, demonstrating compatibility and synergy with monolingual data augmentation.
• The E-step's time-step reward mechanism effectively guides model updates, with high-reward outputs being reinforced and low-quality outputs penalized.
• The method is effective even with limited bilingual data for (Y,Z), showing robustness in low-resource settings.