[Paper Review] Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding

Di Wu, Yiren Chen+2 more|arXiv (Cornell University)|Apr 13, 2021

Topic Modeling26 references4 citations

TL;DR

This paper proposes RoSLU, a domain adaptation-based method that bridges the gap between clean data training and real-world noisy inference in spoken language understanding by jointly embedding clean and noisy samples into a shared vector space and using a denoising generator to correct errors. It achieves state-of-the-art performance on both noisy and clean test sets, with a +3.8 F1-point gain over baselines on the Snips dataset under noisy inference.

ABSTRACT

Spoken language understanding (SLU) system usually consists of various pipeline components, where each component heavily relies on the results of its upstream ones. For example, Intent detection (ID), and slot filling (SF) require its upstream automatic speech recognition (ASR) to transform the voice into text. In this case, the upstream perturbations, e.g. ASR errors, environmental noise and careless user speaking, will propagate to the ID and SF models, thus deteriorating the system performance. Therefore, the well-performing SF and ID models are expected to be noise resistant to some extent. However, existing models are trained on clean data, which causes a extit{gap between clean data training and real-world inference.} To bridge the gap, we propose a method from the perspective of domain adaptation, by which both high- and low-quality samples are embedding into similar vector space. Meanwhile, we design a denoising generation model to reduce the impact of the low-quality samples. Experiments on the widely-used dataset, i.e. Snips, and large scale in-house dataset (10 million training examples) demonstrate that this method not only outperforms the baseline models on real-world (noisy) corpus but also enhances the robustness, that is, it produces high-quality results under a noisy environment. The source code will be released.

Motivation & Objective

• To address the distribution gap between clean data training and real-world noisy inference in industrial SLU systems.
• To enhance model robustness against real-world perturbations such as ASR errors, environmental noise, and careless speech.
• To enable slot filling models to correct in-slot errors—common in real-world inputs but unhandled by traditional models.
• To develop a method that maintains high performance on clean data while significantly improving performance on noisy data.
• To demonstrate the effectiveness of domain adaptation and denoising generation in bridging the real-world inference gap.

Proposed method

• The method employs domain adaptation to map both high-quality clean samples and low-quality noisy samples into a shared, similar vector space.
• A denoising generator model is trained to reconstruct correct slot labels from noisy input sequences, improving robustness to input perturbations.
• The model is trained end-to-end on a combination of clean and synthetic noisy data, enabling it to generalize to real-world noisy inputs.
• The approach uses adversarial training principles to regularize the model and improve generalization under distribution shift.
• The framework is designed to preserve performance on clean data while significantly enhancing robustness on noisy test sets.
• In-slot error correction is enabled by the model’s ability to map semantically similar but noisy slot spans (e.g., 'rock stones' for 'the Rolling Stones') to their correct forms.

Experimental results

Research questions

• RQ1 Can a model trained on both clean and noisy data achieve better robustness in real-world SLU inference without sacrificing performance on clean data?
• RQ2 To what extent can a denoising generator correct in-slot errors caused by ASR or speech variations?
• RQ3 Does domain adaptation between clean and noisy samples effectively bridge the distribution gap in SLU systems?
• RQ4 How does the proposed method compare to strong baselines in terms of F1 score on noisy and clean test sets?
• RQ5 Can the model generalize to large-scale industrial datasets with millions of examples and complex noise patterns?

Key findings

• On the Snips dataset, RoSLU outperforms the best baseline by +3.8 F1 points under noisy inference, demonstrating strong robustness.
• On the in-house dataset with 10 million training examples, RoSLU achieves a +0.64 F1-point gain over the baseline under global (noisy) training, despite a high baseline score.
• The model maintains comparable performance to SOTA models on clean inference, with F1 scores of 91.24 on Snips and 96.01 on in-house data.
• Manual evaluation confirmed that 59 out of 177 sampled in-slot errors were successfully corrected, validating the model’s capacity to handle real-world noise.
• The performance gain from including noisy data in training is more pronounced on smaller datasets like Snips, where noise acts as a regularizer.
• The results show that domain adaptation and denoising generation jointly improve generalization, especially in noisy real-world scenarios.