[Paper Review] CUNI Systems for the WMT22 Czech-Ukrainian Translation Task

Martin Popel, Jindřich Libovický+1 more|arXiv (Cornell University)|Dec 1, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper presents Charles University's submissions to the WMT22 Czech-Ukrainian translation shared task, featuring two constrained systems using block back-translation and tagged back-translation, along with a proprietary system, Charles Translator. Romanization of Ukrainian text had minimal impact on translation quality, while block back-translation slightly outperformed tagged back-translation, with final BLEU scores reaching 35.0 (cs→uk) and 35.8 (uk→cs) on the test set.

ABSTRACT

We present Charles University submissions to the WMT22 General Translation Shared Task on Czech-Ukrainian and Ukrainian-Czech machine translation. We present two constrained submissions based on block back-translation and tagged back-translation and experiment with rule-based romanization of Ukrainian. Our results show that the romanization only has a minor effect on the translation quality. Further, we describe Charles Translator, a system that was developed in March 2022 as a response to the migration from Ukraine to the Czech Republic. Compared to our constrained systems, it did not use the romanization and used some proprietary data sources.

Motivation & Objective

• To develop high-quality machine translation systems for the low-resource Czech-Ukrainian language pair in response to the humanitarian crisis following the Russian invasion of Ukraine.
• To evaluate the effectiveness of rule-based romanization of Ukrainian Cyrillic text in improving cross-lingual transfer between closely related Slavic languages.
• To compare the performance of constrained back-translation techniques (block and tagged) against an unconstrained system trained on proprietary data.
• To assess the impact of data preprocessing, including monolingual data filtering and artificial noise injection, on translation quality.
• To provide a practical, production-ready translation system (Charles Translator) for real-time use in migration support contexts.

Proposed method

• Employed block back-translation using the Marian NMT toolkit, training a sequence-to-sequence model with monolingual data generated by back-translating synthetic target-side sentences.
• Implemented tagged back-translation with special tokens to preserve source language identity during back-translation, using the Tensor2Tensor framework.
• Developed a reversible, rule-based romanization system that maps Ukrainian Cyrillic to Latin script using Czech phonological rules, enabling lexical similarity exploitation.
• Filtered parallel and monolingual data using FastText for language detection, length ratio filtering, and regex-based removal of non-translation elements like emails and proper nouns.
• Applied synthetic noise to source-side sentences to improve robustness, using character-level corruption techniques.
• Trained a lightweight Transformer model on NewsCrawl to restore punctuation and detokenize monolingual Ukrainian data, which had been tokenized and punctuation-removed.

Experimental results

Research questions

• RQ1 Does rule-based romanization of Ukrainian Cyrillic text significantly improve translation quality in low-resource Czech-Ukrainian machine translation?
• RQ2 How do block back-translation and tagged back-translation compare in terms of BLEU, chrF, and COMET scores on the WMT22 test set?
• RQ3 To what extent do data preprocessing choices—such as filtering, noise injection, and monolingual data cleaning—affect system performance?
• RQ4 How does the performance of a constrained system trained on public data compare to an unconstrained system using proprietary data (Charles Translator) in real-world deployment scenarios?
• RQ5 What is the impact of hyperparameters such as batch size on translation quality, especially when compared to architectural or data-augmentation techniques?

Key findings

• Romanization of Ukrainian Cyrillic text had only a minor effect on translation quality, with no consistent improvement across either translation direction.
• Block back-translation achieved slightly better results than tagged back-translation, with a BLEU score of 35.0 (cs→uk) and 35.8 (uk→cs) on the WMT22 test set.
• The unconstrained Charles Translator system, trained on proprietary data and not using romanization, achieved competitive results, with BLEU scores of 34.3 (cs→uk) and 35.9 (uk→cs), comparable to the constrained systems.
• Ensembling multiple models provided only marginal improvements, suggesting that diversity in initialization or data augmentation was limited in impact.
• Batch size had a stronger influence on performance than most data-augmentation or preprocessing techniques; one Cyrillic system with a 3× higher batch size outperformed others despite identical architecture.
• COMET scores were unreliable for this language pair due to lack of training data, making BLEU and chrF more informative metrics for evaluation.