[Paper Review] Simplifying Sentences with Sequence to Sequence Models

Alexander Patrick Mathews, Lexing Xie+1 more|arXiv (Cornell University)|May 15, 2018

Topic Modeling7 references3 citations

TL;DR

This paper proposes S4, a sequence-to-sequence model with a novel word-copy mechanism and custom loss function to improve sentence simplification. By jointly using pre-trained and fine-tuned word embeddings, S4 achieves an 8.8 BLEU score improvement over a baseline, with key gains from word copying (4.9 BLEU points) and embedding tuning (3.8 BLEU points), demonstrating strong performance on semantic preservation and simplification quality.

ABSTRACT

We simplify sentences with an attentive neural network sequence to sequence model, dubbed S4. The model includes a novel word-copy mechanism and loss function to exploit linguistic similarities between the original and simplified sentences. It also jointly uses pre-trained and fine-tuned word embeddings to capture the semantics of complex sentences and to mitigate the effects of limited data. When trained and evaluated on pairs of sentences from thousands of news articles, we observe a 8.8 point improvement in BLEU score over a sequence to sequence baseline; however, learning word substitutions remains difficult. Such sequence to sequence models are promising for other text generation tasks such as style transfer.

Motivation & Objective

• To develop a neural sequence-to-sequence model that simplifies sentences while preserving meaning and improving readability.
• To address the challenge of limited training data in sentence simplification by leveraging pre-trained and fine-tuned word embeddings.
• To enhance word copying in output sequences using a novel feeding mechanism and loss function to reduce reliance on word generation.
• To evaluate the model’s performance on a new sentence-level parallel corpus, focusing on BLEU, ROUGE, and semantic fidelity.
• To investigate the impact of attention, word copying, and embedding composition on simplification quality.

Proposed method

• S4 employs an encoder-decoder architecture with Bahdanov-style attention to align input and output tokens.
• A novel word-copy feeding mechanism enables the model to track and reuse words from the input sentence, improving fidelity during simplification.
• A custom binary cross-entropy loss function is introduced to encourage word copying, reducing the burden on the generator network.
• The model combines pre-trained word embeddings (e.g., from GloVe) with fine-tuned embeddings to enhance semantic understanding of complex sentences.
• The system is trained on a sentence-level parallel corpus derived from news articles, with alignment and simplification quality assessed via automatic metrics.
• An oracle word simplifier is used to isolate the contribution of attention from word substitution errors in ablation studies.

Experimental results

Research questions

• RQ1 How does integrating word-copying mechanisms improve sentence simplification performance in sequence-to-sequence models?
• RQ2 To what extent does combining pre-trained and fine-tuned word embeddings enhance simplification quality under data scarcity?
• RQ3 How do the proposed loss function and feeding mechanism jointly affect BLEU and ROUGE scores?
• RQ4 What is the contribution of attention alignment versus word substitution to overall simplification quality?
• RQ5 How does the model perform in generating semantically accurate and syntactically correct simplifications compared to reference outputs?

Key findings

• The S4 model achieves an 8.8-point improvement in BLEU-4 over a standard sequence-to-sequence baseline.
• The custom loss function (S4+bce) improves BLEU-4 by 4.9 points by encouraging word copying.
• Combining pre-trained and fine-tuned embeddings (S4+gv+bce) leads to an additional 3.8-point BLEU-4 gain.
• Removing the attention mechanism causes a 35.56-point drop in BLEU-4, highlighting its critical role in alignment and semantic retention.
• The S4+gv+bce model produces sentences closest in length to the ground truth, while S4+bce is closest in Flesch readability score.
• When using an oracle word simplifier, BLEU-4 increases by 14.4 points, indicating that word substitution remains a major bottleneck despite model improvements.