[Paper Review] Neural Paraphrase Generation with Stacked Residual LSTM Networks

Aaditya Prakash, Sadid A. Hasan+5 more|arXiv (Cornell University)|Oct 10, 2016

Natural Language Processing TechniquesComputer Science49 references221 citations

TL;DR

The paper introduces a stacked residual LSTM network for paraphrase generation and shows it outperforms various deep-learning baselines on multiple large datasets. Residual connections enable training deeper LSTMs for better paraphrase quality.

ABSTRACT

In this paper, we propose a novel neural approach for paraphrase generation. Conventional para- phrase generation methods either leverage hand-written rules and thesauri-based alignments, or use statistical machine learning principles. To the best of our knowledge, this work is the first to explore deep learning models for paraphrase generation. Our primary contribution is a stacked residual LSTM network, where we add residual connections between LSTM layers. This allows for efficient training of deep LSTMs. We evaluate our model and other state-of-the-art deep learning models on three different datasets: PPDB, WikiAnswers and MSCOCO. Evaluation results demonstrate that our model outperforms sequence to sequence, attention-based and bi- directional LSTM models on BLEU, METEOR, TER and an embedding-based sentence similarity metric.

Motivation & Objective

• Motivate paraphrase generation as a neural sequence-to-sequence task and move beyond rule-based or SMT-based approaches.
• Propose a deep LSTM architecture with residual connections to enable training of deeper encoders/decoders for paraphrase generation.
• Demonstrate the model's effectiveness across large-scale paraphrase datasets on standard evaluation metrics.
• Compare stacked residual LSTM against several baselines including Seq2Seq, attention-based, and bi-directional LSTM models.
• Evaluate robustness across diverse datasets (PPDB, WikiAnswers, MSCOCO).

Proposed method

• Adopt an encoder–decoderSeq2Seq framework for paraphrase generation with LSTM units.
• Introduce stacked residual connections between LSTM layers to enable deeper networks without degradation.
• Use vertical stacking where inputs flow from one layer to the next, with residual connections added after every n layers (n=2).
• Train with SGD, dropout, and perplexity loss; evaluate via beam search during inference (beam sizes 5 and 10).
• Evaluate with BLEU, METEOR, TER, and an embedding-based similarity metric (Emb Greedy).

Experimental results

Research questions

• RQ1 Can residual connections enable effective training of deeper stacked LSTMs for paraphrase generation?
• RQ2 Do stacked residual LSTMs outperform conventional sequence-to-sequence, attention-based, and bidirectional LSTM baselines on large paraphrase datasets?
• RQ3 How does the proposed model perform across varied datasets with different paraphrase characteristics (short phrases vs. longer captions)?
• RQ4 What is the impact of beam size on paraphrase quality for the stacked residual LSTM model?
• RQ5 Is perplexity a suitable loss for training neural paraphrase generators, and how does embedding-based similarity correlate with traditional metrics?

Key findings

• The stacked residual LSTM model consistently improves over baseline models on BLEU and TER across all datasets.
• The model achieves superior Emb Greedy scores on most datasets, with some exceptions at larger beam sizes.
• METEOR gains are dataset-dependent, with the residual LSTM leading on MSCOCO and WikiAnswers, but not always on PPDB.
• Increasing depth (four-layer residual LSTM) yields better performance than shallower configurations.
• Perplexity aligns with model performance, but current metrics can miss semantic and novelty aspects of paraphrase quality.
• The approach works well on diverse data sources (PPDB, WikiAnswers, MSCOCO) without requiring images for MSCOCO in training.