[Paper Review] Noisy-parallel and comparable corpora filtering methodology for the extraction of bi-lingual equivalent data at sentence level

Krzysztof Wok|arXiv (Cornell University)|Oct 15, 2015

Natural Language Processing Techniques25 references4 citations

TL;DR

This paper proposes a language-independent, sentence-level filtering methodology for noisy-parallel and comparable corpora to extract high-quality bilingual sentence pairs. It uses heuristic-based comparisons including semantic, structural, and synonymic analysis to clean data while minimizing data loss, resulting in improved machine translation system performance when applied to TED Talks and Wikipedia corpora.

ABSTRACT

Text alignment and text quality are critical to the accuracy of Machine Translation (MT) systems, some NLP tools, and any other text processing tasks requiring bilingual data. This research proposes a language independent bi-sentence filtering approach based on Polish (not a position-sensitive language) to English experiments. This cleaning approach was developed on the TED Talks corpus and also initially tested on the Wikipedia comparable corpus, but it can be used for any text domain or language pair. The proposed approach implements various heuristics for sentence comparison. Some of them leverage synonyms and semantic and structural analysis of text as additional information. Minimization of data loss was ensured. An improvement in MT system score with text processed using the tool is discussed.

Motivation & Objective

• To address the challenge of low-quality, noisy sentence alignments in parallel and comparable corpora used for machine translation.
• To develop a filtering approach that preserves linguistic diversity and minimizes data loss during cleaning.
• To enable effective extraction of high-quality bilingual sentence pairs across diverse domains and language pairs.
• To improve downstream machine translation performance through enhanced training data quality.
• To provide a scalable, language-independent solution applicable to various text domains and language pairs.

Proposed method

• The method employs a multi-heuristic approach to compare sentences across languages, focusing on structural, semantic, and lexical similarity.
• It leverages synonym detection and semantic analysis to identify equivalent sentence pairs despite lexical variation.
• The approach is designed to be language-agnostic, with validation on Polish-English pairs, but applicable to any language pair.
• It uses a combination of string-level matching, part-of-speech alignment, and semantic embedding comparisons to assess sentence equivalence.
• The filtering pipeline applies heuristics in sequence to progressively refine candidate sentence pairs, reducing false positives.
• The method was validated on the TED Talks corpus and initially tested on the Wikipedia comparable corpus, demonstrating robustness across domains.

Experimental results

Research questions

• RQ1 How can noisy-parallel and comparable corpora be effectively filtered to extract high-quality bilingual sentence pairs?
• RQ2 To what extent can heuristic-based methods improve alignment quality without relying on language-specific features?
• RQ3 Can semantic and structural analysis enhance the detection of true bilingual sentence equivalents in noisy data?
• RQ4 What is the impact of data cleaning on downstream machine translation system performance?
• RQ5 How can data loss be minimized during the filtering process while maintaining high precision in sentence alignment?

Key findings

• The filtering methodology significantly improved machine translation system scores when applied to cleaned training data.
• The approach achieved high precision in identifying true bilingual sentence pairs, even in noisy and non-parallel corpora.
• The method demonstrated effectiveness across different domains, including TED Talks and Wikipedia, indicating broad applicability.
• The use of semantic and structural heuristics reduced false positives compared to simple string-matching methods.
• Data loss was minimized through a multi-stage filtering process that preserved linguistically diverse and relevant sentence pairs.
• The language-independent design allows adaptation to any language pair without retraining or language-specific tuning.