[Paper Review] Boosting Active Learning for Speech Recognition with Noisy Pseudo-labeled Samples

Jihwan Bang, Heesu Kim+2 more|arXiv (Cornell University)|Jun 19, 2020

Speech Recognition and Synthesis42 references4 citations

TL;DR

This paper proposes a training pipeline that enhances active learning in end-to-end speech recognition by incorporating consistency regularization with noisy pseudo-labeled samples from unlabeled data. By applying carefully designed audio augmentations to enforce model consistency across perturbed inputs, the method reduces labeling costs by up to 12.76% in CER under a 1/3 labeling budget, significantly outperforming standard active learning without additional annotation effort.

ABSTRACT

The cost of annotating transcriptions for large speech corpora becomes a bottleneck to maximally enjoy the potential capacity of deep neural network-based automatic speech recognition models. In this paper, we present a new training pipeline boosting the conventional active learning approach targeting label-efficient learning to resolve the mentioned problem. Existing active learning methods only focus on selecting a set of informative samples under a labeling budget. One step further, we suggest that the training efficiency can be further improved by utilizing the unlabeled samples, exceeding the labeling budget, by introducing sophisticatedly configured unsupervised loss complementing supervised loss effectively. We propose new unsupervised loss based on consistency regularization, and we configure appropriate augmentation techniques for utterances to adopt consistency regularization in the automatic speech recognition task. From the qualitative and quantitative experiments on the real-world dataset and under real-usage scenarios, we show that the proposed training pipeline can boost the efficacy of active learning approaches, thus successfully reducing a sustainable amount of human labeling cost.

Motivation & Objective

• To reduce the high human annotation cost in large-scale speech recognition datasets, which limits the scalability of end-to-end ASR models.
• To improve label efficiency in active learning by leveraging unlabeled samples beyond the labeling budget.
• To mitigate performance degradation from noisy pseudo-labels in semi-supervised learning for ASR through robust regularization.
• To enable consistency regularization in ASR by designing audio augmentations that preserve linguistic content while supporting model robustness.
• To validate the proposed pipeline on real-world, heterogeneous-domain speech data under practical, low-budget settings.

Proposed method

• Introduces a hybrid training pipeline combining active learning (HLS) with semi-supervised learning using pseudo-labeled samples (PLS) from unlabeled data.
• Applies consistency regularization (CR) to enforce consistent predictions across original and augmented versions of the same utterance.
• Designs domain-specific audio augmentations (e.g., speed perturbation, noise injection) that preserve phonetic content while enabling effective CR in ASR.
• Uses a two-stage process: first, generate pseudo-labels for high-confidence unlabeled samples; second, apply CR loss to regularize predictions on both original and augmented inputs.
• Implements periodic pseudo-labeling with beam-search decoding at fixed intervals to reduce computational cost while maintaining model freshness.
• Applies a preliminary filtering step (threshold τ = -0.5) to exclude low-confidence pseudo-labels in heterogeneous domain scenarios.

Experimental results

Research questions

• RQ1 Can consistency regularization effectively mitigate performance degradation from noisy pseudo-labels in end-to-end automatic speech recognition?
• RQ2 What types of audio augmentations are suitable for consistency regularization in ASR without corrupting linguistic content?
• RQ3 How does the proposed pipeline improve label efficiency compared to standard active learning under constrained labeling budgets?
• RQ4 To what extent can pseudo-labeled samples from unlabeled data boost model performance when the initial and unlabeled data come from different domains?
• RQ5 What is the optimal frequency of pseudo-labeling to balance training efficiency and model performance?

Key findings

• The proposed +CR-SA pipeline reduces Character Error Rate (CER) by 12.76% compared to standard active learning when the labeling budget is 1/3 of the total unlabeled data.
• Under a 1/10 labeling budget, the method achieves a 4.02% CER improvement over conventional active learning.
• With only 137 hours of labeled data (1/3 of 386 hours), the model achieves performance within 0.82 percentage points of the full-batch model, demonstrating strong label efficiency.
• Consistency regularization significantly reduces the error of pseudo-labels (P-CER), with +CR-Xs showing stable and improving P-CER over training, unlike non-regularized baselines.
• Periodic pseudo-labeling at intervals of 1, 3, or 5 epochs results in only a 0.2 percentage point performance degradation, enabling efficient training with large unlabeled sets.
• In heterogeneous domain settings (e.g., AI Assistant vs. Map Navigation), the +CR-SA-τ pipeline restores model performance close to full-batch performance using only a fraction of labeled data.