[Paper Review] Learning to Jointly Translate and Predict Dropped Pronouns with a Shared Reconstruction Mechanism

Longyue Wang, Zhaopeng Tu+2 more|arXiv (Cornell University)|Oct 15, 2018

Natural Language Processing Techniques12 references4 citations

TL;DR

This paper proposes a shared reconstruction mechanism and joint learning framework to improve neural machine translation of pro-drop languages by simultaneously translating and predicting dropped pronouns (DPs). By sharing a reconstructor across encoder and decoder representations and jointly training DP prediction with translation, the model achieves +1.45 BLEU points over prior work and improves DP prediction F1-score by 9%, significantly reducing error propagation from external predictors.

ABSTRACT

Pronouns are frequently omitted in pro-drop languages, such as Chinese, generally leading to significant challenges with respect to the production of complete translations. Recently, Wang et al. (2018) proposed a novel reconstruction-based approach to alleviating dropped pronoun (DP) translation problems for neural machine translation models. In this work, we improve the original model from two perspectives. First, we employ a shared reconstructor to better exploit encoder and decoder representations. Second, we jointly learn to translate and predict DPs in an end-to-end manner, to avoid the errors propagated from an external DP prediction model. Experimental results show that our approach significantly improves both translation performance and DP prediction accuracy.

Motivation & Objective

• To address the challenge of translating dropped pronouns (DPs) in pro-drop languages like Chinese, where pronouns are omitted but required in non-pro-drop target languages like English.
• To reduce error propagation from external DP prediction models, which currently achieve only 66% F1-score.
• To improve translation quality and DP prediction accuracy by jointly learning translation and DP prediction in an end-to-end manner.
• To enhance representation learning by sharing a reconstructor between encoder and decoder to exploit cross-modal relations.

Proposed method

• A shared reconstructor reads hidden states from both the encoder and decoder to reconstruct the DP-annotated source sentence, enabling knowledge sharing between encoder and decoder representations.
• The model integrates a joint prediction head that predicts DPs at the correct positions during training, using the reconstructor's hidden states as context.
• The training objective includes a reconstruction loss for both the source sentence and the DP prediction, guiding the model to learn DP-relevant features.
• An interaction attention mechanism from encoder to decoder is used to better align DP-related information in the decoder's context.
• The model is trained end-to-end with auxiliary losses for reconstruction and DP prediction, avoiding reliance on external DP prediction models.
• The approach uses attention-based reconstruction with separate context vectors for encoder and decoder representations, computed via attention over the reconstructed source.

Experimental results

Research questions

• RQ1 Can a shared reconstructor that processes both encoder and decoder hidden states improve DP translation performance compared to separate reconstructors?
• RQ2 Does jointly training the NMT model with a DP prediction head reduce error propagation from external DP predictors?
• RQ3 To what extent does joint learning improve both translation quality and DP prediction accuracy?
• RQ4 How does the interaction between encoder and decoder representations affect the model's ability to recover dropped pronouns?
• RQ5 Can the model achieve high performance without relying on external DP prediction tools, even when DPP labeling accuracy is suboptimal?

Key findings

• The proposed model achieves a +1.45 BLEU point improvement over the best result reported by Wang et al. (2018), reaching 34.80 BLEU when using shared reconstruction and joint prediction.
• The jointly trained DP prediction model achieves 75% F1-score, a 9% improvement over the external model’s 66% F1-score.
• The shared reconstructor with interaction attention (enc→dec) achieves the best performance, outperforming the baseline by +3.45 BLEU points.
• When reconstruction is used only during training and not at inference, the model still outperforms the baseline and prior SOTA, indicating strong generalization and practicality.
• The performance gap between automatic and manual DPP labeling is reduced, suggesting that error propagation is mitigated by the joint learning mechanism.
• The model shows a smaller gap from oracle performance (manually labeled DPPs) than prior models, indicating robustness to noisy input DPP annotations.