[Paper Review] Investigation of Error Simulation Techniques for Learning Dialog Policies for Conversational Error Recovery

Maryam Fazel-Zarandi, Longshaokan Marshall Wang+2 more|arXiv (Cornell University)|Nov 8, 2019

Speech and dialogue systems34 references11 citations

TL;DR

This paper investigates audio-level and text-level error simulation techniques for training dialog policies in speech-based virtual assistants, proposing improvements to text-level methods through ASR confidence score prediction and out-of-vocabulary word mapping. It finds that text-level simulation performs comparably to audio-level simulation in realism and is preferred for its speed and simplicity, with improved metrics in word error rate, confidence score distribution, and discriminability from real errors.

ABSTRACT

Training dialog policies for speech-based virtual assistants requires a plethora of conversational data. The data collection phase is often expensive and time consuming due to human involvement. To address this issue, a common solution is to build user simulators for data generation. For the successful deployment of the trained policies into real world domains, it is vital that the user simulator mimics realistic conditions. In particular, speech-based assistants are heavily affected by automatic speech recognition and language understanding errors, hence the user simulator should be able to simulate similar errors. In this paper, we review the existing error simulation methods that induce errors at audio, phoneme, text, or semantic level; and conduct detailed comparisons between the audio-level and text-level methods. In the process, we improve the existing text-level method by introducing confidence score prediction and out-of-vocabulary word mapping. We also explore the impact of audio-level and text-level methods on learning a simple clarification dialog policy to recover from errors to provide insight on future improvement for both approaches.

Motivation & Objective

• To evaluate and compare the realism of audio-level and text-level error simulation techniques for training dialog policies in speech-based assistants.
• To improve existing text-level error simulation by introducing ASR confidence score prediction and out-of-vocabulary word mapping.
• To assess the impact of both simulation methods on learning a clarification-based dialog policy for error recovery.
• To develop a more robust discriminator for evaluating simulation quality by incorporating predicted confidence scores and word error rate features.
• To validate simulation quality through held-out data evaluation and user studies on policy performance.

Proposed method

• Adopts a comparative framework using audio-level simulation (based on real ASR output) and text-level simulation (based on n-gram confusion matrices from training data).
• Improves text-level simulation by training a model to predict ASR confidence scores from simulated ASR outputs, enabling realistic signal modeling without audio input.
• Introduces out-of-vocabulary word mapping using phonetic and semantic similarity to enhance error realism in text-level simulation.
• Enhances the discriminator network used to distinguish real vs. simulated data by including predicted confidence scores and word error rate (WER) as input features.
• Employs deep reinforcement learning with Dueling Double DQN to train a clarification policy, using experience replay and epsilon-greedy exploration with linear decay.
• Conducts ablation studies and user evaluations on held-out data to compare policy performance under different simulation conditions.

Experimental results

Research questions

• RQ1 How do audio-level and text-level error simulation methods compare in terms of word error rate distribution and ASR confidence score realism?
• RQ2 Can text-level error simulation achieve comparable realism to audio-level simulation in mimicking real-world ASR and NLU errors?
• RQ3 To what extent do confidence score prediction and OOV word mapping improve the quality of text-level error simulation?
• RQ4 How do the simulated errors affect the learning of a clarification-based dialog policy in terms of task success and user satisfaction?
• RQ5 Can a discriminator trained on predicted confidence scores and WER features effectively distinguish real from simulated dialog data?

Key findings

• Text-level error simulation achieves word error rate and confidence score distributions that are statistically indistinguishable from real ASR errors, making it nearly as realistic as audio-level simulation.
• The proposed confidence score prediction model successfully estimates ASR confidence from simulated text outputs, enabling more realistic policy training without requiring audio signals.
• Out-of-vocabulary word mapping using phonetic and semantic similarity significantly improves error realism in text-level simulation, especially for rare or domain-specific terms.
• The improved discriminator, incorporating predicted confidence scores and WER, achieves higher accuracy in distinguishing real from simulated data, validating the enhanced simulation quality.
• Policy learning under text-level simulation achieves comparable performance to audio-level simulation in both held-out data evaluation and user studies, with no significant difference in user-rated dialog quality.
• The two-way ANOVA on user study ratings shows a non-significant interaction effect (p = 0.076), suggesting that both simulation methods yield similar user-perceived quality in dialog performance.