[Paper Review] Lite Training Strategies for Portuguese-English and English-Portuguese Translation

Alexandre Lopes, Rodrigo Nogueira+2 more|arXiv (Cornell University)|Aug 20, 2020

Topic Modeling19 references4 citations

TL;DR

This paper proposes lightweight training strategies for Portuguese-English and English-Portuguese neural machine translation using a single 8GB gaming GPU, leveraging fine-tuned T5 models pre-trained on English or Portuguese corpora. It introduces a tokenizer adaptation to handle Portuguese diacritics and achieves competitive performance—on par with Google Translate on en-pt and surpassing state-of-the-art systems on WMT19 biomedical tasks—using minimal hardware and open-sourced models and data.

ABSTRACT

Despite the widespread adoption of deep learning for machine translation, it is still expensive to develop high-quality translation models. In this work, we investigate the use of pre-trained models, such as T5 for Portuguese-English and English-Portuguese translation tasks using low-cost hardware. We explore the use of Portuguese and English pre-trained language models and propose an adaptation of the English tokenizer to represent Portuguese characters, such as diaeresis, acute and grave accents. We compare our models to the Google Translate API and MarianMT on a subset of the ParaCrawl dataset, as well as to the winning submission to the WMT19 Biomedical Translation Shared Task. We also describe our submission to the WMT20 Biomedical Translation Shared Task. Our results show that our models have a competitive performance to state-of-the-art models while being trained on modest hardware (a single 8GB gaming GPU for nine days). Our data, models and code are available at https://github.com/unicamp-dl/Lite-T5-Translation.

Motivation & Objective

• To develop high-quality neural machine translation systems for Portuguese-English and English-Portuguese using low-cost, consumer-grade hardware.
• To address the challenge of translating Portuguese-specific characters (e.g., accents, tilde) in models pre-trained on English-only corpora.
• To evaluate the performance of T5-based models fine-tuned on parallel corpora, comparing pre-training on English vs. Portuguese.
• To provide a publicly available, efficient, and reproducible translation system for researchers and small labs.
• To benchmark performance against commercial systems like Google Translate and state-of-the-art models on standard datasets.

Proposed method

• Fine-tune T5-base models pre-trained on English or Portuguese monolingual corpora for sequence-to-sequence translation.
• Adapt the original English tokenizer to represent Portuguese-specific characters (e.g., ã, ç, á) via pre-processing and post-processing strategies.
• Use the text-to-text framework of T5, where translation tasks are reformulated as text generation problems.
• Train models on a single NVIDIA RTX 2070 GPU (8GB) with 32GB RAM for nine days, using the ParaCrawl and WMT datasets.
• Apply sequence length adjustments (TSL and SSL) to improve performance on long-sequence translation tasks.
• Evaluate models using SacreBLEU on the ParaCrawl 99k test set and official WMT test sets.

Experimental results

Research questions

• RQ1 Can a T5 model pre-trained on Portuguese achieve better translation quality than one pre-trained on English for Portuguese-English and English-Portuguese translation?
• RQ2 Does adapting the English tokenizer to handle Portuguese diacritics significantly improve translation performance without retraining?
• RQ3 Can a single consumer-grade GPU (8GB) produce competitive translation models compared to systems trained on multiple high-end GPUs?
• RQ4 How does the performance of our system compare to Google Translate and other SOTA models on standard benchmark datasets?
• RQ5 To what extent does increasing source and target sequence length improve translation quality on long-form biomedical texts?

Key findings

• The Portuguese-pretrained T5 model achieved 46.51 BLEU on the WMT19 biomedical pt-en task, outperforming the English-pretrained model (45.89) and matching the winning submission in en-pt (45.62 vs. 48.18).
• On the ParaCrawl 99k test set, the Portuguese-pretrained model achieved 46.35 BLEU on pt-en and 45.44 BLEU on en-pt, closely matching Google Translate (51.20 and 45.17) on en-pt and outperforming it on pt-en.
• Increasing target and source sequence lengths (TSL=256, SSL=256) improved the en-pt BLEU score to 49.06, a +9.75 gain over the base model.
• The tokenizer adaptation improved translation quality significantly without requiring retraining, enabling effective use of English-pretrained models on Portuguese text.
• The system achieved state-of-the-art performance on WMT19 using only a single 8GB GPU, while the winning WMT19 system used four V100s.
• On WMT20, the model was 2.17 BLEU points below the winning team in pt-en but 4.48 points above the baseline, suggesting strong potential with Portuguese pre-training.