[Paper Review] Top-down Tree Structured Decoding with Syntactic Connections for Neural Machine Translation and Parsing

Jetic Gū, Hassan S. Shavarani+1 more|arXiv (Cornell University)|Sep 6, 2018

Natural Language Processing Techniques35 references4 citations

TL;DR

This paper proposes Seq2DRNN, a top-down tree-structured neural machine translation model using DRNNs for decoding with syntax-aware attention and syntactic connections (SynC) to improve fluency and reordering. It achieves state-of-the-art translation quality and competitive constituency parsing performance by jointly modeling translation and phrase-structure syntax.

ABSTRACT

The addition of syntax-aware decoding in Neural Machine Translation (NMT) systems requires an effective tree-structured neural network, a syntax-aware attention model and a language generation model that is sensitive to sentence structure. We exploit a top-down tree-structured model called DRNN (Doubly-Recurrent Neural Networks) first proposed by Alvarez-Melis and Jaakola (2017) to create an NMT model called Seq2DRNN that combines a sequential encoder with tree-structured decoding augmented with a syntax-aware attention model. Unlike previous approaches to syntax-based NMT which use dependency parsing models our method uses constituency parsing which we argue provides useful information for translation. In addition, we use the syntactic structure of the sentence to add new connections to the tree-structured decoder neural network (Seq2DRNN+SynC). We compare our NMT model with sequential and state of the art syntax-based NMT models and show that our model produces more fluent translations with better reordering. Since our model is capable of doing translation and constituency parsing at the same time we also compare our parsing accuracy against other neural parsing models.

Motivation & Objective

• To improve neural machine translation (NMT) fluency and reordering by incorporating target-side constituency parsing structures into the decoding process.
• To address the limitation of existing syntax-aware NMT models that rely on bottom-up decoding or dependency parsing by introducing a top-down, tree-structured decoder.
• To explore the utility of constituency parsing (phrase structure) over dependency parsing in guiding NMT decoding and improving translation quality.
• To develop a unified model that performs both neural machine translation and constituency parsing simultaneously with competitive accuracy.
• To introduce syntactic connections (SynC) that enhance the DRNN decoder by integrating structural syntactic information directly into the hidden state transitions.

Proposed method

• Uses a bidirectional LSTM encoder to generate contextual representations of the source sentence.
• Employs a top-down tree-structured decoder based on Doubly-Recurrent Neural Networks (DRNN) to generate target sequences in a hierarchical, phrase-structure-aware manner.
• Introduces a syntax-aware attention mechanism that leverages constituency parse trees to guide attention over source representations.
• Adds syntactic connections (SynC) to the DRNN decoder, enabling direct propagation of syntactic structure information through additional recurrent connections.
• Trains the model end-to-end using standard NMT objectives with cross-entropy loss, while jointly optimizing for translation and parsing performance.
• Evaluates parsing performance using standard constituency parsing metrics (precision, recall, F1) on both in-domain (IWSLT) and out-of-domain (Penn Treebank) datasets.

Experimental results

Research questions

• RQ1 Can a top-down, tree-structured decoder with constituency parsing improve neural machine translation quality compared to standard sequential and syntax-aware models?
• RQ2 Does incorporating syntactic connections (SynC) into a DRNN-based decoder enhance translation fluency and reordering capabilities?
• RQ3 Can a single model jointly perform high-quality neural machine translation and constituency parsing without requiring separate parsing heads?
• RQ4 How does constituency-based syntax compare to dependency-based syntax in guiding NMT decoding, particularly in terms of structural coherence and word reordering?
• RQ5 To what extent does the inclusion of SynC connections improve parsing accuracy while maintaining strong translation performance?

Key findings

• Seq2DRNN+SynC achieved a 96.16 F1 score on unlabelled constituency parsing (IWSLT), outperforming the baseline Seq2DRNN (96.90 F1) on unlabelled spans, indicating strong parsing performance.
• The model achieved a 52.16% exact match rate on sentences with fewer than 40 words, indicating high structural fidelity in translation output.
• Seq2DRNN+SynC produced more fluent translations than the strong Seq2Seq baseline and several state-of-the-art syntax-aware NMT models, particularly in reordering complex syntactic structures.
• The model achieved a 90.5 F1 score on parsing longer sentences, showing robustness to length variation and strong generalization.
• The inclusion of SynC slightly reduced parsing F1 (96.16 vs. 96.90) but significantly improved translation quality and reordering, indicating a favorable trade-off for NMT.
• The model demonstrated competitive performance on Penn Treebank constituency parsing (F1: 87.04), confirming its capability as a joint NMT and parsing system.