[Paper Review] Natural Language Generation in Dialogue using Lexicalized and Delexicalized Data

Shikhar Sharma, Jing He+3 more|arXiv (Cornell University)|Feb 17, 2017

Speech and dialogue systems15 citations

TL;DR

This paper proposes a hybrid neural network model for task-oriented dialogue generation that jointly learns from both lexicalized (slot-value pairs) and delexicalized (slot-only) data using an encoder-decoder framework with attention. By leveraging both data types and fine-tuning with a pretrained sentence autoencoder, the model generates more grammatically correct and natural-sounding responses, as confirmed by human evaluation.

ABSTRACT

Natural language generation plays a critical role in spoken dialogue systems. We present a new approach to natural language generation for task-oriented dialogue using recurrent neural networks in an encoder-decoder framework. In contrast to previous work, our model uses both lexicalized and delexicalized components i.e. slot-value pairs for dialogue acts, with slots and corresponding values aligned together. This allows our model to learn from all available data including the slot-value pairing, rather than being restricted to delexicalized slots. We show that this helps our model generate more natural sentences with better grammar. We further improve our model’s performance by transferring weights learnt from a pretrained sentence auto-encoder. Human evaluation of our best-performing model indicates that it generates sentences which users find more appealing.

Motivation & Objective

• To improve natural language generation in task-oriented dialogue systems by combining lexicalized and delexicalized training data.
• To address the limitation of prior models that rely solely on delexicalized slots, which may reduce linguistic naturalness.
• To enhance grammatical quality and fluency of generated responses through joint learning of slot-value pairs.
• To explore the benefits of transfer learning by initializing the model with weights from a pretrained sentence autoencoder.
• To evaluate the model's performance through human evaluation, focusing on perceived naturalness and appeal of generated utterances.

Proposed method

• The model employs an encoder-decoder architecture with an attention mechanism to generate responses from dialogue acts.
• It incorporates both lexicalized data (slot-value pairs) and delexicalized data (slots only), allowing the model to learn from all available training examples.
• Slot-value pairs are explicitly aligned during training to preserve semantic relationships between slots and their values.
• The model is initialized with fine-tuned weights from a pretrained sentence autoencoder to improve generalization and fluency.
• The architecture uses recurrent neural networks (LSTM or GRU) for sequence modeling in both encoder and decoder components.
• The training objective optimizes for both fluency and correctness by minimizing cross-entropy loss on the generated output.

Experimental results

Research questions

• RQ1 Can joint learning from lexicalized and delexicalized data improve the fluency and grammatical correctness of generated dialogue responses?
• RQ2 How does incorporating slot-value pair alignment affect the model’s ability to generate natural-sounding utterances?
• RQ3 To what extent does transfer learning from a pretrained sentence autoencoder enhance generation quality in dialogue systems?
• RQ4 Does the hybrid data approach outperform models trained exclusively on delexicalized data in human evaluations?
• RQ5 How do human evaluators rate the naturalness and appeal of responses generated by the proposed model compared to baselines?

Key findings

• The model generates responses with better grammar and greater linguistic naturalness compared to models trained only on delexicalized data.
• Human evaluation confirms that the best-performing model produces utterances users find more appealing and fluent.
• The integration of lexicalized data improves the model’s ability to preserve slot-value relationships in generated text.
• Transfer learning from a pretrained sentence autoencoder leads to measurable improvements in response quality.
• The model achieves superior performance by leveraging both types of training data, avoiding the limitations of delexicalization alone.