[Paper Review] Neural Machine Translation with Dynamic Graph Convolutional Decoder

Lei Li, Kai Fan+3 more|arXiv (Cornell University)|May 28, 2023

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes a novel end-to-end neural machine translation framework, Dyn-STGCD, that jointly generates target translations and their syntactic dependency graphs in an auto-regressive manner. By dynamically constructing the target syntactic graph during decoding using a customized graph convolutional decoder, the model improves translation performance across multiple benchmarks, achieving state-of-the-art results on WMT 2014 En-De and En-Fr tasks with BLEU scores of 28.1 and 41.8, respectively.

ABSTRACT

Existing wisdom demonstrates the significance of syntactic knowledge for the improvement of neural machine translation models. However, most previous works merely focus on leveraging the source syntax in the well-known encoder-decoder framework. In sharp contrast, this paper proposes an end-to-end translation architecture from the (graph \& sequence) structural inputs to the (graph \& sequence) outputs, where the target translation and its corresponding syntactic graph are jointly modeled and generated. We propose a customized Dynamic Spatial-Temporal Graph Convolutional Decoder (Dyn-STGCD), which is designed for consuming source feature representations and their syntactic graph, and auto-regressively generating the target syntactic graph and tokens simultaneously. We conduct extensive experiments on five widely acknowledged translation benchmarks, verifying that our proposal achieves consistent improvements over baselines and other syntax-aware variants.

Motivation & Objective

• To address the gap in modeling explicit syntactic structures on the target side in neural machine translation.
• To enable the decoder to dynamically construct and utilize target-side syntactic graphs during auto-regressive generation.
• To improve translation quality by jointly modeling target tokens and their syntactic dependencies.
• To demonstrate that explicit target-side syntactic modeling enhances translation performance beyond source-side syntax only.
• To provide a generalizable graph-to-graph generation framework applicable to other sequence-to-graph tasks.

Proposed method

• Proposes a Dynamic Spatial-Temporal Graph Convolutional Decoder (Dyn-STGCD) that processes both source token representations and their syntactic graph.
• Introduces a dynamic graph construction mechanism where new nodes (predicted tokens) and edges are incrementally added at each decoding step.
• Employs spatial-temporal GCN layers to model both local neighborhood information and sequential dependencies in the target graph.
• Uses a transformer-based encoder to extract contextual representations from the source sentence and its syntactic graph.
• Designs a joint generation objective that optimizes both target token prediction and syntactic graph structure simultaneously.
• Leverages a stand-alone parser to generate pseudo-reference graphs for evaluating the consistency of predicted syntactic structures.

Experimental results

Research questions

• RQ1 Can explicit modeling of target-side syntactic structures improve neural machine translation performance?
• RQ2 Is it feasible to dynamically construct a syntactic graph during auto-regressive decoding using graph convolutional layers?
• RQ3 How does joint generation of target tokens and their syntactic graphs compare to models that only use source-side syntax?
• RQ4 To what extent do the predicted syntactic graphs align with gold-standard parses from external parsers?
• RQ5 Can the proposed Dyn-STGCD framework be generalized beyond translation to other sequence-to-graph generation tasks?

Key findings

• Dyn-STGCD achieves a BLEU score of 28.1 on the WMT 2014 English-German translation task, outperforming the baseline Transformer and other syntax-aware models.
• On the English-French task, Dyn-STGCD achieves a BLEU score of 41.8, surpassing the baseline Transformer (41.0) and other syntax-aware variants.
• The model generates syntactic graphs that show high consistency with outputs from a standalone dependency parser, indicating reliable structural predictions.
• Empirical results on NIST Chinese-English and WMT benchmarks confirm consistent improvements over multiple baselines and syntax-aware models.
• The approach significantly boosts translation quality without using common performance-enhancing techniques like model ensembling or compound splitting.
• The dynamic graph construction mechanism enables the decoder to adaptively incorporate syntactic information during generation, leading to better contextual modeling.