[Paper Review] Automatic TM Cleaning through MT and POS Tagging: Autodesk's Submission to the NLP4TM 2016 Shared Task

Alena Zwahlen, O. Carnal+1 more|arXiv (Cornell University)|May 19, 2016

Natural Language Processing Techniques11 references3 citations

TL;DR

This paper presents a language-independent machine learning approach to detect incorrect entries in translation memories by combining neural machine translation (MT) evaluation metrics—BLEU and Levenshtein distance—and part-of-speech (POS) tagging features. The system, which uses a Random Forest classifier with 9 key features including MT-based similarity and POS alignment, achieved the highest F1 score in the Binary Classification (II) task for English–Italian and English–Spanish, ranking first in both pairs.

ABSTRACT

We describe a machine learning based method to identify incorrect entries in translation memories. It extends previous work by Barbu (2015) through incorporating recall-based machine translation and part-of-speech-tagging features. Our system ranked first in the Binary Classification (II) task for two out of three language pairs: English-Italian and English-Spanish.

Motivation & Objective

• To develop a language-independent method for identifying incorrect translation memory (TM) entries to improve machine translation and human translation efficiency.
• To extend Barbu (2015) by incorporating advanced MT evaluation metrics and POS tagging to enhance detection of non-corresponding or erroneous translations.
• To evaluate the system on real-world TM data across multiple language pairs, focusing on binary classification of correct vs. incorrect entries.
• To assess the impact of domain mismatch on performance, particularly for English–German, where results were weaker than for other language pairs.
• To validate the method in production environments by testing its ability to clean noisy TMs and improve downstream translation quality.

Proposed method

• The system uses a Random Forest classifier trained on a combination of 9 features, including language detection, word and character ratio, and MT-based similarity metrics.
• It applies recall-based MT features using n-best translations to account for lexical variation, such as synonyms, improving robustness to paraphrased source-target pairs.
• BLEU and character-based Levenshtein distance are used as alternative similarity measures to cosine similarity, capturing n-gram and partial word overlaps more effectively.
• POS tagging is performed per language and mapped to 11 language-independent grammatical categories (Petrov et al., 2011) to compare syntactic structure alignment between source and target segments.
• Features such as punctuation similarity and capitalization difference are included to detect markup or formatting errors.
• The model is trained exclusively on shared task data, with hyperparameters tuned via 5-fold cross-validation to maximize weighted F1-score on training sets.

Experimental results

Research questions

• RQ1 Can MT-based evaluation metrics (BLEU and Levenshtein) outperform traditional cosine similarity in detecting incorrect TM entries?
• RQ2 To what extent do POS tag alignments across languages improve the detection of non-corresponding or malformed translations?
• RQ3 Does incorporating n-best MT hypotheses (recall-based features) reduce false positives on valid but paraphrased translations?
• RQ4 How does domain mismatch (e.g., medical/news vs. software) affect the performance of a language-independent TM cleaning system?
• RQ5 Can a feature set combining MT, POS, and linguistic heuristics achieve state-of-the-art results in binary TM cleaning without language-specific tuning?

Key findings

• The system achieved the highest F1 score in the Binary Classification (II) task for English–Italian (F1 = 0.90) and English–Spanish (F1 = 0.89), ranking first in both.
• For English–German, the system achieved a lower F1 (0.79), suggesting possible performance degradation due to domain mismatch or language-specific challenges.
• The inclusion of n-best MT hypotheses and BLEU/Levenshtein metrics improved recall and F1 compared to cosine similarity alone, particularly in handling paraphrased or variant translations.
• POS similarity features (pos_sim_all) contributed significantly to distinguishing correct from incorrect TUs, especially in closely related languages.
• The system outperformed Barbu’s (2015) SVM and Random Forest baselines on all language pairs except English–German, where performance was comparable.
• In the Fine-Grained Classification task, the same feature set underperformed, indicating that the selected features may not be sufficient to distinguish 'almost correct' from 'correct' entries.