[Paper Review] Data Augmentation for Training Dialog Models Robust to Speech Recognition Errors

Longshaokan Wang, Maryam Fazel-Zarandi+3 more|arXiv (Cornell University)|Jun 10, 2020

Topic Modeling14 references4 citations

TL;DR

This paper proposes a data augmentation method that improves dialog model robustness to automatic speech recognition (ASR) errors by simulating ASR noise using a confusion-matrix-based error simulator. The approach enhances model performance—especially in low-resource settings—without modifying model architectures or introducing inference latency.

ABSTRACT

Speech-based virtual assistants, such as Amazon Alexa, Google assistant, and Apple Siri, typically convert users' audio signals to text data through automatic speech recognition (ASR) and feed the text to downstream dialog models for natural language understanding and response generation. The ASR output is error-prone; however, the downstream dialog models are often trained on error-free text data, making them sensitive to ASR errors during inference time. To bridge the gap and make dialog models more robust to ASR errors, we leverage an ASR error simulator to inject noise into the error-free text data, and subsequently train the dialog models with the augmented data. Compared to other approaches for handling ASR errors, such as using ASR lattice or end-to-end methods, our data augmentation approach does not require any modification to the ASR or downstream dialog models; our approach also does not introduce any additional latency during inference time. We perform extensive experiments on benchmark data and show that our approach improves the performance of downstream dialog models in the presence of ASR errors, and it is particularly effective in the low-resource situations where there are constraints on model size or the training data is scarce.

Motivation & Objective

• Address the challenge of dialog models being sensitive to ASR errors during inference, despite being trained on error-free text.
• Overcome limitations of existing methods that require ASR lattice access, model retraining, or additional inference latency.
• Enable robustness to ASR errors in low-resource scenarios with limited training data or constrained model size.
• Develop a plug-and-play data augmentation solution that works with any pre-existing dialog model and ASR system.
• Demonstrate effectiveness across diverse model architectures, including small and lightweight models.

Proposed method

• Use a confusion-matrix-based ASR error simulator to generate synthetic ASR hypotheses by injecting common word-level errors into clean reference text.
• Construct n-gram confusion matrices from aligned ASR hypotheses and reference texts to model frequent word substitutions and errors.
• Apply the error simulator during training to augment clean text data with realistic ASR-like variations.
• Train downstream dialog models (e.g., for intent classification) on both original and augmented data to improve robustness.
• Preserve model architecture and inference pipeline unchanged, avoiding any latency increase during inference.
• Support both scenarios with and without existing ASR hypotheses, enabling broad applicability to diverse training data sources.

Experimental results

Research questions

• RQ1 Can data augmentation using a simulated ASR error generator improve dialog model robustness to real-world ASR errors?
• RQ2 How effective is the proposed method in low-resource settings with limited training data or small model architectures?
• RQ3 Does the method maintain performance improvements across different dialog model architectures, including lightweight models?
• RQ4 How does the method compare to approaches requiring ASR lattice or end-to-end integration in terms of inference efficiency?
• RQ5 Can the method be combined with other techniques like acoustic embeddings or multi-task learning for further gains?

Key findings

• The proposed data augmentation method significantly improves dialog model performance under ASR errors, with a 20.18% absolute improvement in accuracy when training on only 1% of the data using BERT.
• On a simple neural network (Simple NN), the method achieves a 35.20% absolute improvement in accuracy when training on 5% of the data, demonstrating strong gains in low-data regimes.
• For lightweight models (GloVe + 1-layer LSTM), the method yields a 3.89% absolute improvement in S1 and 2.34% in S2, showing effectiveness even without large pre-trained embeddings.
• The performance gains are most pronounced in low-resource settings, with the largest improvements observed on the smallest training subsets.
• The method remains effective even when the model is simplified, indicating strong practical value for latency- or size-constrained deployments.
• The approach is complementary to other techniques like acoustic embeddings and multi-task learning, suggesting potential for further gains when combined.