[Paper Review] Lingua Custodia's participation at the WMT 2021 Machine Translation using Terminologies shared task

Melissa Ailem, Jinghsu Liu+1 more|arXiv (Cornell University)|Nov 3, 2021

Natural Language Processing Techniques4 citations

TL;DR

This paper presents Lingua Custodia's approach to constrained neural machine translation in the WMT 2021 shared task, using training-time data augmentation and constraint token masking to improve terminology adherence. The method significantly boosts exact-match accuracy to 91.9% on English-French translation while maintaining high BLEU (44.90) and COMET (0.681) scores, outperforming both standard Transformers and inference-time constrained decoders.

ABSTRACT

This paper describes Lingua Custodia's submission to the WMT21 shared task on machine translation using terminologies. We consider three directions, namely English to French, Russian, and Chinese. We rely on a Transformer-based architecture as a building block, and we explore a method which introduces two main changes to the standard procedure to handle terminologies. The first one consists in augmenting the training data in such a way as to encourage the model to learn a copy behavior when it encounters terminology constraint terms. The second change is constraint token masking, whose purpose is to ease copy behavior learning and to improve model generalization. Empirical results show that our method satisfies most terminology constraints while maintaining high translation quality.

Motivation & Objective

• To improve neural machine translation models' adherence to terminology constraints in specialized domains such as biomedicine and public health.
• To address the challenge of low terminology coverage in parallel training data by leveraging monolingual data for data augmentation.
• To enhance model generalization and robustness when handling conflicting terminology constraints through a novel masking strategy.
• To develop a training-time approach that avoids inference-time computational overhead while ensuring strict terminology compliance.

Proposed method

• Augmenting training data by tagging source-side terminology terms with <S> and </S> tags to guide the model toward copy behavior.
• Introducing a masked token strategy where the source term is replaced with a MASK token after the <S> tag to teach the model to copy the target term.
• Using a Transformer-based architecture with tied embeddings and 8 attention heads, trained on multilingual data with BPE or SentencePiece tokenization.
• Applying BPE subword segmentation with 40k merges for English-French and English-Russian, and SentencePiece for English-Chinese, resulting in vocabulary sizes between 39,996 and 52,172.
• Training models for up to 100 epochs with early stopping based on validation loss, using a beam size of 5 during inference.
• Evaluating the system using standard MT metrics (BLEU, COMET) and terminology-specific metrics (Exact-Match Accuracy, Window Overlap, 1-TERm).

Experimental results

Research questions

• RQ1 Can training-time data augmentation with terminology tagging improve model adherence to lexical constraints in neural machine translation?
• RQ2 Does constraint token masking enhance model generalization and robustness when handling conflicting terminology mappings?
• RQ3 How does the proposed method compare to inference-time constrained decoding in terms of terminology accuracy and translation quality?
• RQ4 To what extent can monolingual data be leveraged to increase the number of training examples containing terminology terms?
• RQ5 Can the method maintain high translation quality (BLEU, COMET) while achieving near-perfect terminology adherence?

Key findings

• The TAG+MASK model achieved 91.9% exact-match accuracy on the English-to-French test set, significantly outperforming the standard Transformer (32.5%) and constrained decoder (85.6%) baselines.
• On the English-to-French set, the model reached a BLEU score of 44.90 and a COMET score of 0.681, demonstrating high translation quality alongside strong terminology compliance.
• For English-to-Russian, the system achieved 84.9% exact-match accuracy and a BLEU score of 29.13, showing strong performance on a morphologically rich language.
• For English-to-Chinese, the model obtained 82.9% exact-match accuracy and a BLEU score of 29.16, indicating effective adaptation to low-resource, non-Latin script translation.
• The Window Overlap (2) and (3) metrics showed consistent improvements across all language pairs, with 34.4% and 33.5% overlap respectively on English-French, indicating better contextual placement of terminology terms.
• The 1-TERm score of 0.598 on English-French indicates a substantial reduction in terminology-related edit errors compared to the baseline.