[Paper Review] On Improving Informativity and Grammaticality for Multi-Sentence Compression

Elaheh ShafieiBavani, Mohammad Ebrahimi+2 more|arXiv (Cornell University)|May 7, 2016

Natural Language Processing Techniques28 references3 citations

TL;DR

This paper proposes a novel word graph-based approach to multi-sentence compression that enhances both informativity and grammaticality by integrating Multiword Expressions (MWE) merging, synonymy-based word mapping, and re-ranking using a 7-gram POS-based language model (POS-LM). The method achieves state-of-the-art performance on a newswire dataset, improving both automatic and human evaluation scores while maintaining a high compression ratio.

ABSTRACT

Multi Sentence Compression (MSC) is of great value to many real world applications, such as guided microblog summarization, opinion summarization and newswire summarization. Recently, word graph-based approaches have been proposed and become popular in MSC. Their key assumption is that redundancy among a set of related sentences provides a reliable way to generate informative and grammatical sentences. In this paper, we propose an effective approach to enhance the word graph-based MSC and tackle the issue that most of the state-of-the-art MSC approaches are confronted with: i.e., improving both informativity and grammaticality at the same time. Our approach consists of three main components: (1) a merging method based on Multiword Expressions (MWE); (2) a mapping strategy based on synonymy between words; (3) a re-ranking step to identify the best compression candidates generated using a POS-based language model (POS-LM). We demonstrate the effectiveness of this novel approach using a dataset made of clusters of English newswire sentences. The observed improvements on informativity and grammaticality of the generated compressions show that our approach is superior to state-of-the-art MSC methods.

Motivation & Objective

• To address the longstanding challenge in multi-sentence compression (MSC) of simultaneously improving both informativity and grammaticality.
• To overcome the trade-off observed in prior word graph-based MSC methods, where enhancing one quality often degrades the other.
• To improve the quality of compressed sentences by leveraging linguistic structures such as MWEs and synonymy.
• To develop a re-ranking strategy using a POS-based language model that favors grammatically correct and fluent compressions.
• To demonstrate the effectiveness of the proposed method on a standard English newswire dataset through both automatic and human evaluation.

Proposed method

• Merging words in Multiword Expressions (MWEs) into single nodes within the word graph to reduce ambiguity and improve semantic coherence.
• Applying synonymy by replacing MWEs with their one-word synonyms when available and using synonyms of individual words to guide node mapping.
• Employing a 7-gram POS-based language model (POS-LM) to re-rank the k-shortest paths in the word graph, favoring grammatically well-formed compressions.
• Constructing a word graph from POS-tagged sentences and stopwords, using dynamic programming to find k-shortest paths as candidate compressions.
• Using a hybrid strategy combining MWE merging, synonym mapping, and POS-LM re-ranking to generate high-quality compressions.
• Validating results via human annotation and automatic metrics (Rouge, Bleu), with manual MWE detection and synonym alignment using WordNet.

Experimental results

Research questions

• RQ1 Can merging Multiword Expressions (MWEs) into single nodes in the word graph improve the informativity of compressed sentences?
• RQ2 Does incorporating synonymy between words and MWEs enhance the quality of multi-sentence compressions?
• RQ3 Can a POS-based language model (POS-LM) effectively re-rank compression candidates to improve grammaticality without sacrificing informativity?
• RQ4 To what extent do the individual components—MWE merging, synonym mapping, and POS-LM re-ranking—contribute to the overall improvement in compression quality?
• RQ5 How well do automatic evaluation metrics (Rouge and Bleu) correlate with human judgments of informativity and grammaticality?

Key findings

• The proposed method achieved a Rouge-1 F1 score of 0.5841, Rouge-2 of 0.4284, Rouge-su4 of 0.3950, and Bleu-4 of 0.6913, outperforming both baseline methods.
• Human evaluation showed a significant improvement in grammaticality (1.47) and informativity (1.36) on diverse clusters, with a compression rate of 48%.
• The POS-LM re-ranking step contributed most to grammaticality, as evidenced by a 0.6913 Bleu-4 score, which correlates strongly with grammatical fluency.
• MWE merging and synonym mapping significantly boosted informativity, as shown by higher Rouge scores, particularly in Rouge-1 and Rouge-2.
• The combination of all three components (MWE, synonymy, POS-LM) yielded the best overall performance, with a 0.5–1.0 correlation between automatic metrics and human ratings.
• The study confirmed that Rouge correlates better with informativity, while Bleu correlates better with grammaticality, validating their use as complementary evaluation metrics.