[Paper Review] Data Augmentation for Spoken Language Understanding via Pretrained Models

Baolin Peng, Chenguang Zhu+2 more|arXiv (Cornell University)|Apr 29, 2020

Topic Modeling23 references21 citations

TL;DR

This paper proposes a data augmentation framework for spoken language understanding (SLU) using pretrained language models to address data scarcity, particularly in two underexplored scenarios: rich-in-ontology and rich-in-utterance settings. The method generates diverse, high-quality synthetic training utterances, significantly improving SLU model performance across multiple benchmarks.

ABSTRACT

The training of spoken language understanding (SLU) models often faces the problem of data scarcity. In this paper, we put forward a data augmentation method with pretrained language models to boost the variability and accuracy of generated utterances. Furthermore, we investigate and propose solutions to two previously overlooked scenarios of data scarcity in SLU: i) Rich-in-Ontology: ontology information with numerous valid dialogue acts are given; ii) Rich-in-Utterance: a large number of unlabelled utterances are available. Empirical results show that our method can produce synthetic training data that boosts the performance of language understanding models in various scenarios.

Motivation & Objective

• To address data scarcity in spoken language understanding (SLU) by generating synthetic training data.
• To investigate and resolve two overlooked data scarcity scenarios: rich-in-ontology and rich-in-utterance.
• To improve the variability and accuracy of generated utterances using pretrained language models.
• To enhance SLU model performance through effective data augmentation in low-resource settings.

Proposed method

• Leverage pretrained language models to generate synthetic utterances conditioned on dialogue acts and ontology information.
• Design a conditional generation strategy that preserves semantic and structural consistency with the original ontology.
• Utilize unlabelled utterances in the rich-in-utterance scenario to guide generation via fine-tuning or prompt engineering.
• Apply back-translation or paraphrasing techniques to increase utterance diversity while maintaining intent and slot accuracy.
• Integrate generated data into the training pipeline to fine-tune SLU models.
• Evaluate the quality of synthetic data through automatic and human evaluation metrics.

Experimental results

Research questions

• RQ1 Can pretrained language models effectively generate diverse and accurate utterances for SLU in low-resource settings?
• RQ2 How does data augmentation perform in the rich-in-ontology scenario, where ontology is comprehensive but utterances are scarce?
• RQ3 How does the method perform when abundant unlabelled utterances are available but no labels exist (rich-in-utterance)?
• RQ4 What impact does synthetic data have on downstream SLU model performance across different data scarcity scenarios?
• RQ5 How does the quality of generated utterances compare to real data in terms of intent and slot prediction accuracy?

Key findings

• The proposed method significantly improves SLU model performance by generating high-quality synthetic utterances that enhance model generalization.
• In the rich-in-ontology scenario, the model achieves higher intent and slot accuracy compared to baseline data augmentation techniques.
• In the rich-in-utterance scenario, the method effectively leverages unlabelled data to generate diverse, semantically valid utterances.
• Empirical results show consistent performance gains across multiple SLU benchmarks, demonstrating robustness across different data scarcity conditions.
• The generated synthetic data maintains high semantic fidelity and is effective in reducing data scarcity without requiring additional annotations.