[Paper Review] Improving Code-Switching and Named Entity Recognition in ASR with Speech Editing based Data Augmentation

Liang Zheng, Zheshu Song+4 more|arXiv (Cornell University)|Jun 14, 2023

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes a speech editing-based data augmentation method to improve code-switching and named entity recognition (NER) in end-to-end automatic speech recognition (ASR). By editing real speech using text-based speech editing models, the approach generates more coherent, diverse, and realistic augmented audio than traditional audio splicing or neural TTS methods, leading to significant improvements in WER and named entity recall/precision on both in-domain and out-of-domain test sets.

ABSTRACT

Recently, end-to-end (E2E) automatic speech recognition (ASR) models have made great strides and exhibit excellent performance in general speech recognition. However, there remain several challenging scenarios that E2E models are not competent in, such as code-switching and named entity recognition (NER). Data augmentation is a common and effective practice for these two scenarios. However, the current data augmentation methods mainly rely on audio splicing and text-to-speech (TTS) models, which might result in discontinuous, unrealistic, and less diversified speech. To mitigate these potential issues, we propose a novel data augmentation method by applying the text-based speech editing model. The augmented speech from speech editing systems is more coherent and diversified, also more akin to real speech. The experimental results on code-switching and NER tasks show that our proposed method can significantly outperform the audio splicing and neural TTS based data augmentation systems.

Motivation & Objective

• To address the challenge of poor performance in end-to-end ASR models for code-switching and named entity recognition due to data scarcity and long-tail distributions of rare words.
• To overcome limitations of existing data augmentation techniques—audio splicing and neural TTS—by reducing acoustic discontinuities, artifacts, and lack of diversity in synthesized speech.
• To improve model generalization and robustness on unseen code-switched and named entity utterances by generating more realistic and coherent augmented training data.
• To evaluate the effectiveness of speech editing-based data augmentation on both code-switching and NER tasks in real-world ASR scenarios.

Proposed method

• The method applies a text-based speech editing model to modify real speech audio by inserting or replacing target words (e.g., named entities or code-switched segments) while preserving the original speaker’s voice and acoustic characteristics.
• The approach generates augmented speech by editing existing real audio samples, ensuring continuity and naturalness, unlike audio splicing which concatenates disjoint segments.
• The method produces two variants: one with full audio editing (Aug_edit) and another with only acoustic feature editing (Aug_edit-feats), allowing for trade-offs between realism and training efficiency.
• A hybrid data augmentation strategy combines edited speech with neural TTS-generated speech (Aug_edit-feats+tts) to further increase diversity and improve generalization.
• The augmented data is used to fine-tune a pre-trained ASR model, with the encoder’s lower 11 layers frozen to maintain general speech recognition capacity while adapting to target domains.
• The training data is mixed with real, non-target speech to balance domain shift and prevent overfitting to synthesized data.

Experimental results

Research questions

• RQ1 Can speech editing-based data augmentation outperform audio splicing and neural TTS in improving code-switching recognition in end-to-end ASR?
• RQ2 Does speech editing-based data augmentation enhance named entity recognition performance in both in-domain and out-of-domain settings?
• RQ3 How does the coherence and acoustic realism of edited speech compare to that of audio spliced or TTS-generated speech in real-world ASR applications?
• RQ4 What is the impact of combining edited speech with neural TTS-generated speech on model generalization and robustness?

Key findings

• The proposed speech editing-based data augmentation (Aug_edit) achieved the lowest word error rate (WER) and the highest named entity recall and precision on both in-domain and out-of-domain test sets compared to audio splicing and neural TTS baselines.
• The hybrid approach (Aug_edit-feats+tts) achieved the best overall performance, outperforming all other methods on all metrics, including WER and named entity recognition scores.
• Fine-tuning with mixed real and augmented data significantly improved model generalization, reducing overfitting compared to using synthesized data alone.
• The Aug_edit-feats variant achieved better balance between WER and named entity recognition performance than Aug_edit, Aug_splice, and Aug_tts, due to preserved acoustic consistency in unmodified regions.
• The results demonstrate that speech editing generates more realistic and coherent augmented speech than audio splicing or neural TTS, leading to measurable gains in ASR robustness for rare and out-of-vocabulary words.
• The performance gains were consistent across both code-switching and named entity recognition tasks, validating the method’s broad applicability to data-scarce ASR scenarios.