[Paper Review] Enriching Non-Autoregressive Transformer with Syntactic and SemanticStructures for Neural Machine Translation

Ye Liu, Yao Wan+3 more|arXiv (Cornell University)|Jan 22, 2021

Natural Language Processing Techniques28 references4 citations

TL;DR

This paper proposes SNAT, a non-autoregressive Transformer for neural machine translation that integrates syntactic (POS) and semantic (NER) structure labels into the model to improve translation quality and efficiency. By incorporating structural guidance and intermediate latent alignment, SNAT achieves 9.3x faster decoding than autoregressive Transformers while matching or exceeding state-of-the-art non-autoregressive models in BLEU score on WMT14 En-De and WMT16 En-Ro.

ABSTRACT

The non-autoregressive models have boosted the efficiency of neural machine translation through parallelized decoding at the cost of effectiveness when comparing with the autoregressive counterparts. In this paper, we claim that the syntactic and semantic structures among natural language are critical for non-autoregressive machine translation and can further improve the performance. However, these structures are rarely considered in the existing non-autoregressive models. Inspired by this intuition, we propose to incorporate the explicit syntactic and semantic structures of languages into a non-autoregressive Transformer, for the task of neural machine translation. Moreover, we also consider the intermediate latent alignment within target sentences to better learn the long-term token dependencies. Experimental results on two real-world datasets (i.e., WMT14 En-De and WMT16 En-Ro) show that our model achieves a significantly faster speed, as well as keeps the translation quality when compared with several state-of-the-art non-autoregressive models.

Motivation & Objective

• To address the performance gap between non-autoregressive and autoregressive NMT models by incorporating syntactic and semantic structures.
• To mitigate the multimodality issue in non-autoregressive decoding through explicit structural supervision.
• To improve long-range dependency modeling via intermediate latent alignment in decoder layers.
• To achieve high-speed inference without sacrificing translation quality.

Proposed method

• The model takes both input tokens and their POS/NER tags as input, enabling the encoder to attend to syntactic and semantic structure.
• Syntactic and semantic structure labels are embedded and concatenated with word embeddings to guide the decoder.
• Intermediate latent alignment is introduced by regularizing attention maps between decoder layers and target tokens to model long-term dependencies.
• The model uses a multi-task learning objective combining standard cross-entropy loss and alignment regularization.
• The architecture is end-to-end trainable with standard Transformer components, including multi-head attention and feed-forward networks.
• The method is evaluated on WMT14 En-De and WMT16 En-Ro, with ablation studies on structure usage and alignment layers.

Experimental results

Research questions

• RQ1 Can integrating POS and NER tags into a non-autoregressive Transformer improve translation quality?
• RQ2 How does intermediate latent alignment between decoder layers affect long-range dependency modeling?
• RQ3 Does the proposed model maintain high speed while improving translation performance compared to state-of-the-art non-autoregressive models?
• RQ4 How does the model perform across varying sentence lengths, especially long sequences?

Key findings

• SNAT achieves a 9.3x speedup over the autoregressive Transformer on WMT14 En-De with batch size 1, while maintaining competitive BLEU scores.
• Incorporating POS tags improves BLEU by 1.37 points on WMT14 En-De, and NER tags by 1.25 points, demonstrating the value of syntactic and semantic supervision.
• Using the third-layer representation (Z³) for intermediate alignment yields the highest improvement (+1.46 BLEU) over the baseline.
• SNAT maintains stable performance across sentence lengths, with BLEU scores decreasing only slightly from 28.67 (length ≤10) to 17.69 (length >100), unlike vanilla NAT which degrades sharply.
• On WMT14 En-De, SNAT achieves 24.57 BLEU with Z³ and both POS/NER tags, outperforming several state-of-the-art NAT models in both speed and accuracy.
• SNAT is faster than DCRF-NAT while achieving better translation quality, confirming its efficiency and effectiveness.