[Paper Review] Large-scale Hierarchical Alignment for Data-driven Text Rewriting

Nikola I. Nikolov, Richard H. R. Hahnloser|arXiv (Cornell University)|Oct 18, 2018

Text Readability and Simplification31 references4 citations

TL;DR

This paper proposes Large-scale Hierarchical Alignment (LHA), an unsupervised method that extracts pseudo-parallel sentence pairs from monolingual corpora of different text styles using hierarchical nearest-neighbor search over pre-trained sentence and document embeddings. The method achieves competitive performance in text simplification even when used alone, demonstrating that large-scale pseudo-parallel data can effectively replace or augment costly parallel corpora.

ABSTRACT

We propose a simple unsupervised method for extracting pseudo-parallel monolingual sentence pairs from comparable corpora representative of two different text styles, such as news articles and scientific papers. Our approach does not require a seed parallel corpus, but instead relies solely on hierarchical search over pre-trained embeddings of documents and sentences. We demonstrate the effectiveness of our method through automatic and extrinsic evaluation on text simplification from the normal to the Simple Wikipedia. We show that pseudo-parallel sentences extracted with our method not only supplement existing parallel data, but can even lead to competitive performance on their own.

Motivation & Objective

• To address the scarcity of parallel monolingual corpora for text rewriting tasks like simplification and style transfer.
• To develop an unsupervised method that does not require any seed parallel data or manual annotation.
• To enable large-scale, efficient, and memory-efficient extraction of semantically aligned sentence pairs from comparable corpora.
• To evaluate whether pseudo-parallel data alone can achieve competitive performance in text rewriting tasks.
• To demonstrate the method's effectiveness on text simplification using automatic and human evaluation.

Proposed method

• LHA performs hierarchical nearest-neighbor search: first aligning documents between two corpora based on pre-trained document embeddings, then aligning sentences within matched document pairs using sentence embeddings.
• The method relies solely on pre-trained sentence and document embeddings without requiring fine-tuning or parallel seed data.
• Sentence pairs are extracted based on high semantic similarity while preserving stylistic differences between the source and target corpora.
• The approach is robust to noise and scales efficiently to corpora with hundreds of millions of sentences.
• Pseudo-parallel pairs are used to pre-train or fine-tune neural machine translation (NMT) models for text simplification.
• The method is evaluated on both automatic alignment benchmarks and extrinsic text simplification tasks using automatic and human evaluation.

Experimental results

Research questions

• RQ1 Can pseudo-parallel sentence pairs be effectively extracted from monolingual comparable corpora without any parallel seed data?
• RQ2 How well does LHA perform in aligning sentences across different text styles, such as news and scientific writing?
• RQ3 Can pseudo-parallel data extracted by LHA alone achieve competitive performance in text simplification compared to models trained on real parallel data?
• RQ4 Does mixing in-domain and out-of-domain pseudo-parallel data improve performance on text simplification tasks?
• RQ5 How do models trained on LHA-generated data compare to baselines in grammaticality, meaning preservation, and simplicity?

Key findings

• LHA successfully extracts high-quality pseudo-parallel sentence pairs from monolingual corpora using only pre-trained embeddings and hierarchical nearest-neighbor search.
• The method achieves competitive performance on text simplification even when trained solely on pseudo-parallel data, outperforming a baseline trained on 272K real parallel sentences in simplicity.
• Human evaluation shows that models trained on mixed pseudo-parallel data (pseudo-all) achieve simplicity scores comparable to the reference Simple Wikipedia, with a mean simplicity rating of 0.77.
• The model trained on pseudo-parallel data alone (pseudo-all) achieved a grammaticality score of 4.02 and meaning preservation of 2.96, indicating acceptable quality despite no real parallel data.
• Incorporating in-domain pseudo-parallel data (wiki-simp-all) alongside out-of-domain data (wiki-news-all) improved performance, showing the benefit of domain-specific alignment.
• The best model, trained on a mixture of parallel and pseudo-parallel data (wiki-simp-65), achieved a meaning preservation score of 3.76, slightly better than the baseline and close to the reference standard.