[Paper Review] Integrating Transformer and Paraphrase Rules for Sentence Simplification

Sanqiang Zhao, Rui Meng+3 more|arXiv (Cornell University)|Oct 26, 2018

Text Readability and Simplification15 references14 citations

TL;DR

This paper proposes a novel neural network model, DMASS+DCSS, that integrates a multi-layer, multi-head attention architecture with an external knowledge base (Simple PPDB) to improve sentence simplification. By combining dedicated memory for simplification rules with internal parameterized rule learning, the model achieves state-of-the-art performance in both SARI score and rule utilization, outperforming existing methods in accuracy and coverage of simplification rules.

ABSTRACT

Sentence simplification aims to reduce the complexity of a sentence while retaining its original meaning. Current models for sentence simplification adopted ideas from ma- chine translation studies and implicitly learned simplification mapping rules from normal- simple sentence pairs. In this paper, we explore a novel model based on a multi-layer and multi-head attention architecture and we pro- pose two innovative approaches to integrate the Simple PPDB (A Paraphrase Database for Simplification), an external paraphrase knowledge base for simplification that covers a wide range of real-world simplification rules. The experiments show that the integration provides two major benefits: (1) the integrated model outperforms multiple state- of-the-art baseline models for sentence simplification in the literature (2) through analysis of the rule utilization, the model seeks to select more accurate simplification rules. The code and models used in the paper are available at https://github.com/ Sanqiang/text_simplification.

Motivation & Objective

• To address the limitations of neural networks in learning low-frequency simplification rules due to insufficient training data.
• To improve sentence simplification performance by integrating human-curated simplification rules from the Simple PPDB knowledge base into neural models.
• To explore whether external memory and internal parameterization of rules can jointly enhance rule recall and precision in sentence simplification.
• To demonstrate that combining memory-augmented and critic-based training approaches leads to superior generalization and rule application accuracy.

Proposed method

• The model employs a multi-layer, multi-head attention mechanism (inspired by Vaswani et al., 2017) to better attend to relevant words in the input sentence for simplification.
• The DMASS approach introduces an external memory component to store and retrieve simplification rules from the Simple PPDB, enabling explicit rule access during decoding.
• The DCSS approach integrates simplification rules directly into the model’s shared parameters via a modified loss function that encourages learning of infrequent rules.
• The combined DMASS+DCSS model leverages both external memory and internal parameterization to balance rule recall and precision.
• The models are trained on normal-simple sentence pairs and evaluated using SARI and rule utilization metrics.
• Beam search with varying sizes is used to analyze the impact on word simplification accuracy and rule application.

Experimental results

Research questions

• RQ1 Can integrating an external paraphrase knowledge base (Simple PPDB) improve the performance of neural sentence simplification models?
• RQ2 Does using dedicated memory for storing simplification rules enhance model performance compared to internal parameterization alone?
• RQ3 How does the integration of explicit rules affect the model’s ability to apply both frequent and infrequent simplification rules?
• RQ4 To what extent does beam search size influence rule utilization and simplification accuracy?
• RQ5 Can a hybrid approach combining external memory and internal rule learning achieve better performance than either method alone?

Key findings

• The DMASS+DCSS model achieves the highest SARI score among all compared models, demonstrating superior overall simplification quality.
• The model shows the best balance between recall and precision in rule utilization, with high coverage of applicable simplification rules and high accuracy in applying them.
• The integration of Simple PPDB significantly improves performance, especially in word addition and deletion operations, which are critical for high SARI scores.
• Larger beam search sizes correlate strongly with improved rule utilization, suggesting better exploration of valid simplification paths.
• DMASS outperforms DCSS in rule utilization, indicating that external memory helps preserve and apply low-frequency rules more effectively.
• SBMT-SARI, despite high recall, performs poorly in precision due to over-simplification, highlighting the benefit of rule-aware memory in reducing incorrect transformations.