[Paper Review] Error Correction in ASR using Sequence-to-Sequence Models

Samrat Dutta, Shreyansh Jain+4 more|arXiv (Cornell University)|Feb 2, 2022

Speech Recognition and Synthesis10 citations

TL;DR

This paper proposes RoBART, a fine-tuned BART model for correcting automatic speech recognition (ASR) errors using synthetic and real ASR errors. By leveraging phoneme sequences and adaptive training on augmented data, it achieves significant WER reduction on accented speech, though it underperforms on grammatical error correction due to limited context capture.

ABSTRACT

Post-editing in Automatic Speech Recognition (ASR) entails automatically correcting common and systematic errors produced by the ASR system. The outputs of an ASR system are largely prone to phonetic and spelling errors. In this paper, we propose to use a powerful pre-trained sequence-to-sequence model, BART, further adaptively trained to serve as a denoising model, to correct errors of such types. The adaptive training is performed on an augmented dataset obtained by synthetically inducing errors as well as by incorporating actual errors from an existing ASR system. We also propose a simple approach to rescore the outputs using word level alignments. Experimental results on accented speech data demonstrate that our strategy effectively rectifies a significant number of ASR errors and produces improved WER results when compared against a competitive baseline. We also highlight a negative result obtained on the related grammatical error correction task in Hindi language showing the limitation in capturing wider context by our proposed model.

Motivation & Objective

• To address the challenge of correcting common, systematic ASR errors such as phonetic confusions, spelling mistakes, and word boundary errors.
• To overcome the lack of supervised training data for ASR error correction by generating synthetic error data.
• To improve ASR output quality through fine-tuning a pre-trained sequence-to-sequence model (BART) on ASR-plausible errors.
• To evaluate the model’s generalization on a related grammatical error correction (GEC) task in Hindi, highlighting its limitations.

Proposed method

• Fine-tune the pre-trained BART model on a combination of synthetic ASR errors and real ASR hypotheses to adapt it to speech-based error patterns.
• Generate synthetic training data by applying phonetically plausible edits—such as character drops, substitutions, and word boundary shifts—to reference transcriptions.
• Incorporate phoneme sequences as an auxiliary input to enhance the model’s ability to detect and correct phonetically driven errors.
• Use word-level alignment to rescore and combine outputs from the ASR system and the RoBART model, improving final transcription quality.
• Adapt the mBART model for grammatical error correction in Hindi, using both random masking and error-focused masking strategies.
• Evaluate performance using WER on accented speech and GLEU on a Hindi GEC dataset to compare with baselines.

Experimental results

Research questions

• RQ1 Can a pre-trained sequence-to-sequence model like BART be effectively fine-tuned to correct common ASR errors using synthetic and real error data?
• RQ2 How does incorporating phoneme sequences as input improve error correction performance in ASR?
• RQ3 Does the proposed data augmentation strategy significantly reduce word error rate (WER) on accented speech data?
• RQ4 Can the same model architecture generalize to grammatical error correction, and if not, why?
• RQ5 What role does context awareness play in distinguishing between phonetic errors (ASR) and morphological errors (GEC)?

Key findings

• The RoBART model achieves significant WER reduction on accented speech data by leveraging fine-tuning on synthetic and real ASR errors.
• Combining ASR hypotheses with RoBART outputs via word-level alignment and rescoring further improves transcription quality.
• The model performs well on phonetically driven errors such as spelling mistakes, character drops, and word boundary confusion.
• On the Hindi grammatical error correction task, the mBART-based model underperforms compared to baselines, particularly on morphological and syntactic errors.
• The failure on GEC is attributed to the model’s inability to capture wider contextual dependencies, despite strong performance on localized lexical errors.
• Error-focused masking during fine-tuning did not improve performance over simple random masking, suggesting that context modeling is more critical than error-specific masking.