[Paper Review] Controlling the Output Length of Neural Machine Translation

Surafel M. Lakew, Mattia Di Gangi+1 more|arXiv (Cornell University)|Oct 23, 2019

Natural Language Processing TechniquesComputer Science36 references33 citations

TL;DR

The paper presents two methods to control NMT output length in a Transformer: (1) length-token conditioning by prepending a length class token, and (2) length-encoding in the decoder’s positional embeddings; both aim to produce translations that are not longer than the source, with trade-offs in BLEU and length accuracy.

ABSTRACT

The recent advances introduced by neural machine translation (NMT) are rapidly expanding the application fields of machine translation, as well as reshaping the quality level to be targeted. In particular, if translations have to fit some given layout, quality should not only be measured in terms of adequacy and fluency, but also length. Exemplary cases are the translation of document files, subtitles, and scripts for dubbing, where the output length should ideally be as close as possible to the length of the input text. This paper addresses for the first time, to the best of our knowledge, the problem of controlling the output length in NMT. We investigate two methods for biasing the output length with a transformer architecture: i) conditioning the output to a given target-source length-ratio class and ii) enriching the transformer positional embedding with length information. Our experiments show that both methods can induce the network to generate shorter translations, as well as acquiring interpretable linguistic skills.

Motivation & Objective

• Motivate why controlling output length is valuable for layout-constrained MT tasks (documents, subtitles, dubbing).
• Investigate how to bias Transformer NMT toward shorter/longer translations via two methods.
• Assess the impact of length-control methods on translation quality (BLEU) and length metrics across data scales.
• Demonstrate that a combined approach can offer controllable length with minimal quality loss.
• Explore practical training strategies (from-scratch vs fine-tuning) for length-control models.

Proposed method

• Introduce three length groups (short, normal, long) by tokenizing the source with a length class token and train a single model to handle all groups.
• Develop a length encoding in the Transformer decoder that represents remaining or proportional target length, using absolute (len-pos) and relative (quantized len/pos) variants.
• Combine both length-token and length-encoding methods to leverage coarse and fine-grained length control.
• Optionally fine-tune a pre-trained NMT model with length information to decouple baseline quality from length control.
• Evaluate using BLEU and BLEU* (to account for brevity) on En-It and En-De; report mean length ratios LR_src and LR_ref.
• Experiment with small (TED MuST-C) and large data conditions to test robustness across data regimes.

Experimental results

Research questions

• RQ1 Can a Transformer NMT model be guided to generate translations of controlled length without large quality losses?
• RQ2 Do length-token conditioning and length-encoding provide complementary benefits for length control?
• RQ3 How does combining both methods affect translation quality and length predictability across language pairs and data scales?
• RQ4 What is the impact of length-control approaches when fine-tuning a pre-trained model versus training from scratch?
• RQ5 How close can the output length be kept to the source length (LR_src ≈ 1.0) while maintaining BLEU scores?

Key findings

• Length-token conditioning yields coarse-grained length control with minimal BLEU loss and can bring LR_src near 1.0 for some languages.
• Length-encoding can achieve finer-grained length control, but absolute encoding may degrade translation quality due to truncation; relative encoding offers a middle ground.
• A combined Tok+Enc approach achieves diverse length styles (short, normal, long) and can tune target length with limited quality degradation.
• Fine-tuning a baseline model with length information often preserves BLEU while enabling length control close to the source length.
• In large data settings, token-short can reduce length to LR_src ≈ 1.00 with BLEU around 34–36 depending on language pair, while normal/token settings maintain higher BLEU.
• Manual evaluation suggests shorter translations incur small quality degradations that are statistically significant but involve recognizable paraphrase and simplification strategies.