[Paper Review] Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems

Tsung-Hsien Wen, Milica Gašić+4 more|arXiv (Cornell University)|Aug 7, 2015

Speech and dialogue systems39 references126 citations

TL;DR

This paper proposes a semantically conditioned LSTM-based neural network for natural language generation in spoken dialogue systems, enabling end-to-end training on unaligned data without handcrafted rules. The model jointly optimizes sentence planning and surface realization using cross-entropy loss, producing fluent, varied, and human-preferred responses across multiple domains.

ABSTRACT

Natural language generation (NLG) is a critical component of spoken dialogue and it has a significant impact both on usability and perceived quality. Most NLG systems in common use employ rules and heuristics and tend to generate rigid and stylised responses without the natural variation of human language. They are also not easily scaled to systems covering multiple domains and languages. This paper presents a statistical language generator based on a semantically controlled Long Short-term Memory (LSTM) structure. The LSTM generator can learn from unaligned data by jointly optimising sentence planning and surface realisation using a simple cross entropy training criterion, and language variation can be easily achieved by sampling from output candidates. With fewer heuristics, an objective evaluation in two differing test domains showed the proposed method improved performance compared to previous methods. Human judges scored the LSTM system higher on informativeness and naturalness and overall preferred it to the other systems.

Motivation & Objective

• To address the limitations of rule-based NLG systems, which produce rigid, repetitive responses and are hard to scale across domains and languages.
• To reduce reliance on handcrafted templates and heuristics in NLG by enabling data-driven learning from unaligned utterance-semantic pairs.
• To improve natural language generation quality by learning linguistic variation and fluency directly from data using a deep neural architecture.
• To enable scalable, multilingual, and domain-adaptable NLG through end-to-end training with minimal human intervention.
• To evaluate the model’s performance against baselines using both objective metrics and human evaluation in two distinct dialogue domains.

Proposed method

• Proposes a semantically conditioned Long Short-Term Memory (SC-LSTM) cell that jointly models sentence planning and surface realization within a single recurrent architecture.
• Trains the model end-to-end using a simple cross-entropy loss function on unaligned training data, eliminating the need for heuristic rules or pre-defined templates.
• Introduces a deep architecture by stacking multiple SC-LSTM layers to improve representational capacity and generation quality.
• Employs a backward reranker to refine generated outputs for improved fluency, using a language model to re-rank candidate sequences.
• Generates diverse responses through stochastic sampling from the output distribution, naturally capturing linguistic variation.
• Uses a joint optimization framework that allows the model to learn semantic-to-text mapping directly from raw utterance-semantic pairs.

Experimental results

Research questions

• RQ1 Can a neural network-based NLG system trained on unaligned data outperform rule-based and template-based systems in fluency and naturalness?
• RQ2 To what extent can a semantically conditioned LSTM model generate linguistically varied responses without explicit heuristic rules?
• RQ3 How does the proposed end-to-end training approach compare to existing statistical and corpus-based NLG methods in objective and subjective evaluation?
• RQ4 Can the model generalize across different domains (e.g., restaurant and hotel information) with minimal reconfiguration?
• RQ5 Does the deep architecture (+deep) improve generation quality compared to its shallow counterpart (sc-lstm) in both automatic and human evaluations?

Key findings

• The SC-LSTM model achieved the best performance on two objective metrics—BLEU and ROUGE—across both the San Francisco restaurant and hotel domains.
• Human judges rated the SC-LSTM system significantly higher than baselines in informativeness and naturalness, with a strong preference for the deep variant (+deep).
• The deep SC-LSTM model (+deep) showed a statistically significant preference in the pairwise preference test, indicating superior perceived quality.
• The model generated diverse, fluent, and contextually appropriate responses, as evidenced by top-5 outputs showing syntactic variation and natural phrasing.
• The system demonstrated strong domain generalization, achieving high performance on two distinct ontologies with minimal architectural changes.
• The absence of heuristics and reliance on end-to-end training enabled faster development and easier extension to new domains and languages.