[Paper Review] A Constrained Sequence-to-Sequence Neural Model for Sentence Simplification

Yaoyuan Zhang, Zhenxu Ye+3 more|arXiv (Cornell University)|Apr 7, 2017

Text Readability and Simplification13 references17 citations

TL;DR

This paper proposes a two-step neural sequence-to-sequence model that combines word-level simplification with constrained sentence-level generation to produce fluent, grammatically correct, and truly simplified sentences. By enforcing constraints on specific simplified words during decoding, the model outperforms strong baselines in both automatic and human evaluations, achieving higher simplicity, fluency, and readability scores.

ABSTRACT

Sentence simplification reduces semantic complexity to benefit people with language impairments. Previous simplification studies on the sentence level and word level have achieved promising results but also meet great challenges. For sentence-level studies, sentences after simplification are fluent but sometimes are not really simplified. For word-level studies, words are simplified but also have potential grammar errors due to different usages of words before and after simplification. In this paper, we propose a two-step simplification framework by combining both the word-level and the sentence-level simplifications, making use of their corresponding advantages. Based on the two-step framework, we implement a novel constrained neural generation model to simplify sentences given simplified words. The final results on Wikipedia and Simple Wikipedia aligned datasets indicate that our method yields better performance than various baselines.

Motivation & Objective

• To address the limitations of existing sentence simplification models that either fail to simplify meaningfully or introduce grammatical errors.
• To combine the strengths of word-level simplification (lexical substitution) and sentence-level simplification (syntactic restructure) for improved output quality.
• To develop a neural generation model that enforces constraints on required simplified words during decoding to ensure semantic fidelity and simplification.
• To improve readability and grammatical correctness while maintaining semantic similarity to the original sentence.

Proposed method

• A two-step simplification framework: first, complex words are replaced with simpler synonyms using a pre-constructed knowledge base (e.g., PPDB).
• Second, a constrained sequence-to-sequence model generates a simplified sentence conditioned on the simplified words, ensuring their presence in the output.
• The model extends from single-word constraints to multiple-word constraints during decoding, using attention mechanisms and pointer-generator networks to preserve input structure.
• The constrained generation process uses a modified decoding strategy that enforces the inclusion of specified simplified words at generation time.
• The model is trained end-to-end on parallel Wikipedia and Simple Wikipedia datasets to optimize for fluency, simplicity, and semantic similarity.
• The approach leverages neural machine translation principles but incorporates hard constraints to avoid generating identical or un-simplified sentences.

Experimental results

Research questions

• RQ1 Can combining word-level and sentence-level simplification improve the quality of simplified sentences in terms of fluency and grammaticality?
• RQ2 How effective is a constrained seq2seq model in ensuring that specific simplified words are included during sentence generation?
• RQ3 Does enforcing constraints on generated words lead to better readability and semantic preservation compared to unconstrained models?
• RQ4 Can a two-step framework reduce the risk of grammatical errors while still achieving meaningful simplification?

Key findings

• The constrained seq2seq model achieved an iBLEU score of 20.26, significantly outperforming baselines such as Moses (99.62 BLEU(O,I) but 0.02 simplicity) and SBMT.
• The model achieved the highest Flesch readability score (FK) of 12.74, indicating better readability than all baselines.
• Human evaluation showed the constrained seq2seq model scored 2.81 in Meaning and 0.96 in Simplicity, outperforming Lexical Substitution (which had high meaning but poor grammar) and other baselines.
• The Multi-Constrained Seq2seq model achieved 19.87 iBLEU and 33.16 SARI, demonstrating strong performance in both fluency and semantic similarity.
• Case studies confirmed that the model correctly inserted simplified words (e.g., 'center' instead of 'hub') and restructured complex phrases, avoiding common grammatical errors from simple substitution.
• The model successfully avoided generating identical sentences to the input, unlike Moses and SBMT, which produced 116 and 99 identical outputs respectively.