[Paper Review] Predicting Target Language CCG Supertags Improves Neural Machine Translation

Maria Nădejde, Siva Reddy+5 more|arXiv (Cornell University)|Feb 3, 2017

Natural Language Processing Techniques27 references4 citations

TL;DR

This paper proposes interleaving CCG supertags with target-side words in the decoder of a neural machine translation (NMT) model to explicitly model target-language syntax. By tightly coupling words and syntax, the method improves translation quality by 0.9 BLEU for German→English and 1.2 BLEU for Romanian→English, outperforming multitask training and showing consistent gains on complex syntactic phenomena like prepositional phrase attachment.

ABSTRACT

Neural machine translation (NMT) models are able to partially learn syntactic information from sequential lexical information. Still, some complex syntactic phenomena such as prepositional phrase attachment are poorly modeled. This work aims to answer two questions: 1) Does explicitly modeling target language syntax help NMT? 2) Is tight integration of words and syntax better than multitask training? We introduce syntactic information in the form of CCG supertags in the decoder, by interleaving the target supertags with the word sequence. Our results on WMT data show that explicitly modeling target-syntax improves machine translation quality for German->English, a high-resource pair, and for Romanian->English, a low-resource pair and also several syntactic phenomena including prepositional phrase attachment. Furthermore, a tight coupling of words and syntax improves translation quality more than multitask training. By combining target-syntax with adding source-side dependency labels in the embedding layer, we obtain a total improvement of 0.9 BLEU for German->English and 1.2 BLEU for Romanian->English.

Motivation & Objective

• To investigate whether explicitly modeling target-language syntax improves neural machine translation quality.
• To compare tight integration of words and syntax (via interleaving) against loose coupling via multitask training.
• To evaluate the complementarity of source-side and target-side syntactic features in NMT.
• To analyze the impact of target syntax on diverse syntactic phenomena and sentence lengths.
• To demonstrate that CCG supertags can be effectively predicted by NMT models, enabling end-to-end syntax-aware translation.

Proposed method

• Introduce CCG supertags—syntactic categories encoding subcategorization, attachments, and morphology—into the decoder by interleaving them with target words in the sequence.
• Train a sequence-to-sequence NMT model that jointly predicts target words and their corresponding CCG supertags in a tightly coupled manner.
• Implement a multitask baseline where the model shares encoder and decoder components but trains separately on word translation and supertag prediction.
• Combine the proposed target-syntax method with Sennrich and Haddow’s (2016) source-side linguistic feature framework, integrating dependency labels and POS tags in the embedding layer.
• Use the CCGBank for CCG supertag annotation and evaluate supertag prediction accuracy to validate the model’s syntactic understanding.
• Evaluate performance on WMT datasets for German→English and Romanian→English, measuring BLEU and analyzing gains across sentence length and syntactic phenomena.

Experimental results

Research questions

• RQ1 Does explicitly modeling target-language syntax improve neural machine translation quality?
• RQ2 Is tight integration of words and syntax—via interleaving supertags—more effective than loose coupling through multitask training?
• RQ3 How does target-syntax integration affect translation quality for low-resource and high-resource language pairs?
• RQ4 To what extent do source-side and target-side syntactic features complement each other in NMT?
• RQ5 Does the integration of target syntax improve performance on specific syntactic phenomena such as prepositional phrase attachment and clause embedding?

Key findings

• Interleaving CCG supertags with target words in the decoder improves BLEU by 0.9 points for German→English and 1.2 BLEU points for Romanian→English, outperforming both baseline NMT and multitask training.
• The tight coupling of words and syntax via interleaving yields greater improvements than multitask training, which loosely couples word and supertag prediction.
• Combining target-syntax with source-side dependency labels increases translation quality more than either signal alone, demonstrating complementary benefits.
• Improvements are most significant for longer sentences (over 35 words), particularly those with complex syntactic structures like subordinate and coordinate clauses.
• The model achieves high CCG supertag prediction accuracy (95.6% for German→English and 93.2% for Romanian→English), indicating reliable syntactic signal learning.
• Fine-grained analysis confirms consistent gains on prepositional phrase attachment and other syntactic phenomena, with notable improvements on long and structurally complex sentences.