[Paper Review] Schema-Guided Natural Language Generation

Yuheng Du, Shereen Oraby+6 more|arXiv (Cornell University)|May 11, 2020

Topic Modeling29 references4 citations

TL;DR

This paper introduces Schema-Guided Natural Language Generation (SG-NLG), a novel task that enhances neural text generation by leveraging rich, contextual schemata—including intent and slot descriptions—instead of simple slot-value MRs. Using a re-purposed DSTC8 dataset, the authors show that models trained with schema information produce higher-quality, more diverse, and semantically accurate outputs, with GPT-2 achieving the highest human evaluation scores (3.61/5) and improved generalization across seen and unseen domains.

ABSTRACT

Neural network based approaches to data-to-text natural language generation (NLG) have gained popularity in recent years, with the goal of generating a natural language prompt that accurately realizes an input meaning representation. To facilitate the training of neural network models, researchers created large datasets of paired utterances and their meaning representations. However, the creation of such datasets is an arduous task and they mostly consist of simple meaning representations composed of slot and value tokens to be realized. These representations do not include any contextual information that an NLG system can use when trying to generalize, such as domain information and descriptions of slots and values. In this paper, we present the novel task of Schema-Guided Natural Language Generation (SG-NLG). Here, the goal is still to generate a natural language prompt, but in SG-NLG, the input MRs are paired with rich schemata providing contextual information. To generate a dataset for SG-NLG we re-purpose an existing dataset for another task: dialog state tracking, which includes a large and rich schema spanning multiple different attributes, including information about the domain, user intent, and slot descriptions. We train different state-of-the-art models for neural natural language generation on this dataset and show that in many cases, including rich schema information allows our models to produce higher quality outputs both in terms of semantics and diversity. We also conduct experiments comparing model performance on seen versus unseen domains, and present a human evaluation demonstrating high ratings for overall output quality.

Motivation & Objective

• To address the limitation of existing neural NLG datasets that use only basic slot-value meaning representations without contextual or domain-level information.
• To improve generalization and output quality in data-to-text generation by incorporating rich schema information such as intent descriptions, slot types, and domain context.
• To investigate whether schema-guided generation enables better performance on both seen and unseen domains, particularly in zero-shot or few-shot settings.
• To develop and evaluate a new dataset for SG-NLG by re-purposing dialog state tracking data with detailed schema annotations.
• To compare state-of-the-art models (Seq2Seq, CVAE, GPT-2) in the context of schema-guided generation and assess their strengths in semantics, fluency, and diversity.

Proposed method

• The authors construct a new dataset by re-purposing the DSTC8 dialog state tracking dataset, which includes rich schemata with intent descriptions, slot descriptions, domain information, and surface utterances.
• The input to the generation model is a turn-level schema that includes the speaker, dialog act, slots, values, and descriptive metadata such as 'CUISINE: "Cuisine of food served in the restaurant"'.
• Three state-of-the-art models—Sequence-to-Sequence (Seq2Seq), Conditional Variational Autoencoder (CVAE), and GPT-2—are fine-tuned on the SG-NLG dataset to generate delexicalized system prompts from the schemata.
• The models are evaluated using automatic metrics (BLEU, METEOR, SER), diversity (bigram vocabulary size), and human evaluation on semantics, grammar, naturalness, and overall quality.
• The authors perform zero-shot evaluation by testing models on unseen domains, assessing generalization capability.
• Constrained decoding and schema integration are used to guide generation toward accurate content realization while maintaining fluency.

Experimental results

Research questions

• RQ1 Can incorporating rich schema information such as intent and slot descriptions improve the semantic accuracy and fluency of neural text generation models?
• RQ2 How does schema-guided generation affect model generalization across seen versus unseen domains?
• RQ3 Do different neural generation architectures (Seq2Seq, CVAE, GPT-2) exhibit distinct strengths when trained on schema-rich data?
• RQ4 To what extent does schema guidance improve output diversity and reduce semantic errors compared to standard MR-based generation?
• RQ5 Can human evaluation confirm that schema-guided outputs are perceived as higher quality than those from standard MR-based models?

Key findings

• GPT-2 achieved the highest human evaluation score of 3.61 out of 5, closely approaching the reference quality of 3.97, indicating strong overall output quality.
• The use of rich schema information improved semantic exactness, with the lowest average Semantic Error Rate (SER) of 0.18 among the models evaluated.
• GPT-2 showed the highest diversity, achieving a 2.6K bigram vocabulary size, significantly outperforming Seq2Seq and CVAE.
• BLEU and METEOR scores improved by up to 0.43 and 0.61 respectively when using schema-guided training, indicating better reference similarity.
• CVAE models produced fluent and natural outputs but had the highest SER (0.32), indicating frequent semantic inaccuracies despite strong fluency.
• Models fine-tuned on schema-guided data generalized well to unseen domains, demonstrating the potential for zero-shot or few-shot adaptation.