[Paper Review] Towards Robust and Generalizable Training: An Empirical Study of Noisy Slot Filling for Input Perturbations

Jiachi Liu, Liwen Wang+9 more|arXiv (Cornell University)|Oct 5, 2023

Speech and dialogue systemsComputer Science3 citations

TL;DR

This paper introduces Noise-SF, a human-annotated benchmark dataset with five realistic noise types (typos, speech errors, paraphrasing, verbosity, simplification) for evaluating slot filling robustness. It proposes a universal noise-robust training framework combining text-level and feature-level data augmentation, demonstrating that character-level methods like CharAug and continuous interpolation improve performance on different noise types, with significant F1 gains on mixed-noise scenarios.

ABSTRACT

In real dialogue scenarios, as there are unknown input noises in the utterances, existing supervised slot filling models often perform poorly in practical applications. Even though there are some studies on noise-robust models, these works are only evaluated on rule-based synthetic datasets, which is limiting, making it difficult to promote the research of noise-robust methods. In this paper, we introduce a noise robustness evaluation dataset named Noise-SF for slot filling task. The proposed dataset contains five types of human-annotated noise, and all those noises are exactly existed in real extensive robust-training methods of slot filling into the proposed framework. By conducting exhaustive empirical evaluation experiments on Noise-SF, we find that baseline models have poor performance in robustness evaluation, and the proposed framework can effectively improve the robustness of models. Based on the empirical experimental results, we make some forward-looking suggestions to fuel the research in this direction. Our dataset Noise-SF will be released at https://github.com/dongguanting/Noise-SF.

Motivation & Objective

• To address the lack of realistic, human-annotated noisy datasets for evaluating slot filling robustness in real dialogue systems.
• To investigate the effectiveness of various data augmentation strategies in improving model robustness against diverse input noise types.
• To develop a universal noise-robust training framework applicable to slot filling models for improved generalization.
• To provide empirical insights and forward-looking guidance for future research in noise-robust NLP models.

Proposed method

• Constructed Noise-SF by extracting and manually annotating utterances with five realistic noise types from the RADDLE dataset.
• Proposed a universal noise-robust training framework integrating five text-level and four feature-level augmentation techniques.
• Applied text-level augmentations such as character-level corruption (e.g., CharAug) and sentence-level paraphrasing to generate robust training samples.
• Integrated feature-level augmentation via continuous interpolation to enhance model consistency under input perturbations.
• Conducted exhaustive ablation studies on noise type, augmentation type, and combination modes to evaluate robustness.
• Built a multi-noise mixed dataset to evaluate generalization under complex, real-world noise combinations.

Experimental results

Research questions

• RQ1 How do different types of real-world input noise (e.g., typos, speech errors, paraphrasing) affect the performance of supervised slot filling models?
• RQ2 How effective are various text-level and feature-level data augmentation methods in improving robustness across different noise types?
• RQ3 What is the optimal combination of text-level and feature-level augmentation for maximizing robustness in mixed-noise scenarios?
• RQ4 Can the proposed framework generalize across diverse noise distributions and improve model generalization in real-world dialogue systems?

Key findings

• Baseline models trained on clean data suffer severe performance degradation on Noise-SF, with F1 dropping from 95.8 to as low as 64.3 on typo noise.
• Text-level augmentation methods like CharAug show superior performance on character-level noise, improving robustness in typo and speech error scenarios.
• Feature-level augmentation via continuous interpolation yields better results on sentence-level noise such as paraphrasing and verbosity.
• Combined use of text-level and feature-level augmentations leads to the strongest robustness, especially in multi-noise mixed scenarios.
• The proposed framework achieves consistent performance gains across all noise types, validating its effectiveness and generalizability.
• Simulation experiments confirm that the framework maintains robustness even when multiple noise types are present simultaneously.