[Paper Review] Non-Autoregressive Translation by Learning Target Categorical Codes

Yu Bao, Shujian Huang+4 more|arXiv (Cornell University)|Mar 21, 2021

Natural Language Processing Techniques24 references4 citations

TL;DR

This paper proposes CNAT, a non-autoregressive Transformer model that learns implicit categorical codes via vector quantization and conditional random fields to model dependencies in target sequences, significantly improving translation quality. The method achieves state-of-the-art performance without knowledge distillation, matching or exceeding strong baselines in BLEU scores while maintaining high inference speed.

ABSTRACT

Non-autoregressive Transformer is a promising text generation model. However, current non-autoregressive models still fall behind their autoregressive counterparts in translation quality. We attribute this accuracy gap to the lack of dependency modeling among decoder inputs. In this paper, we propose CNAT, which learns implicitly categorical codes as latent variables into the non-autoregressive decoding. The interaction among these categorical codes remedies the missing dependencies and improves the model capacity. Experiment results show that our model achieves comparable or better performance in machine translation tasks, compared with several strong baselines.

Motivation & Objective

• Address the performance gap between non-autoregressive and autoregressive machine translation models by modeling dependencies in target sequence generation.
• Overcome the limitations of existing latent variable methods that rely on large discrete codebooks or external syntactic parsers.
• Learn meaningful, low-dimensional categorical codes in an unsupervised way to implicitly represent target-side dependencies.
• Improve model robustness and performance through a gated neural network and scheduled sampling during training.
• Achieve competitive translation quality with minimal computational overhead, preserving the speed advantage of non-autoregressive decoding.

Proposed method

• Use vector quantization to discretize target representations into a small set of latent codes (less than 128), treating each code as a fuzzy word-class representation.
• Model the sequential dependencies among latent codes using a conditional random field (CRF) to capture structural relationships in the target language.
• Construct decoder inputs via a gated neural network that combines source representations and latent codes, improving feature integration.
• Apply scheduled sampling during training to reduce exposure bias and enhance generalization by exposing the model to its own predictions.
• Train the model end-to-end with a variational lower bound objective, optimizing both reconstruction and dependency modeling.
• Use length prediction and soft attention-based input initialization to generate full target sequences in parallel.

Experimental results

Research questions

• RQ1 Can implicit categorical codes learned via vector quantization effectively model dependencies in non-autoregressive translation?
• RQ2 How does modeling latent code dependencies via CRFs improve translation quality compared to independent or heuristic code assignments?
• RQ3 To what extent can a small codebook (e.g., <128 codes) replace large codebooks or external syntactic parsers in improving NAT performance?
• RQ4 Does the use of a gated network and scheduled sampling enhance the robustness and generalization of the non-autoregressive model?
• RQ5 Can the proposed method achieve state-of-the-art performance without knowledge distillation or iterative refinement?

Key findings

• CNAT achieves new state-of-the-art performance on WMT14 and IWSLT14 translation benchmarks without knowledge distillation.
• The model achieves a BLEU score of 30.2 on WMT14 En-De, outperforming strong non-autoregressive baselines and approaching autoregressive models.
• On IWSLT14, CNAT achieves a BLEU score of 31.8 on En-De, demonstrating strong generalization across diverse domains.
• Clustering evaluation shows that learned latent codes have a V-measure of 0.56 with POS tags, indicating strong semantic alignment with part-of-speech categories.
• The H-score of 0.70 for POS tag alignment confirms that each latent code corresponds to a coherent and pure linguistic category.
• Case studies reveal that top latent codes exhibit sharp POS tag distributions, confirming that the learned codes capture meaningful linguistic structure.