[Paper Review] Improving Readability for Automatic Speech Recognition Transcription

Junwei Liao, Şefik Emre Eskimez+6 more|arXiv (Cornell University)|Apr 9, 2020

Natural Language Processing Techniques54 references19 citations

TL;DR

This paper introduces ASR Post-processing for Readability (APR), a novel NLP task that transforms noisy, disfluent ASR transcripts into fluent, grammatically correct, and human-readable text using fine-tuned pre-trained models. By synthesizing 1.1 million APR training samples from GEC datasets via TTS and ASR, the authors show that RoBERTa-based models significantly outperform traditional pipelines, especially in fluency and readability-aware metrics.

ABSTRACT

Modern Automatic Speech Recognition (ASR) systems can achieve high performance in terms of recognition accuracy. However, a perfectly accurate transcript still can be challenging to read due to grammatical errors, disfluency, and other errata common in spoken communication. Many downstream tasks and human readers rely on the output of the ASR system; therefore, errors introduced by the speaker and ASR system alike will be propagated to the next task in the pipeline. In this work, we propose a novel NLP task called ASR post-processing for readability (APR) that aims to transform the noisy ASR output into a readable text for humans and downstream tasks while maintaining the semantic meaning of the speaker. In addition, we describe a method to address the lack of task-specific data by synthesizing examples for the APR task using the datasets collected for Grammatical Error Correction (GEC) followed by text-to-speech (TTS) and ASR. Furthermore, we propose metrics borrowed from similar tasks to evaluate performance on the APR task. We compare fine-tuned models based on several open-sourced and adapted pre-trained models with the traditional pipeline method. Our results suggest that finetuned models improve the performance on the APR task significantly, hinting at the potential benefits of using APR systems. We hope that the read, understand, and rewrite approach of our work can serve as a basis that many NLP tasks and human readers can benefit from.

Motivation & Objective

• To address the gap in readability of ASR transcripts, which remain difficult to understand despite high recognition accuracy.
• To propose a new NLP task—ASR Post-processing for Readability (APR)—that jointly corrects errors and enhances fluency in a single step.
• To overcome the lack of task-specific data by synthesizing realistic APR training examples using GEC datasets, TTS, and ASR pipelines.
• To evaluate APR performance using a combination of metrics, including readability-aware WER, BLEU, M2, and GLEU, to better reflect human readability.

Proposed method

• Synthesized 1.1 million APR training samples by converting grammatically incorrect sentences from GEC datasets into speech using a TTS system, then transcribing them via an ASR system to create noisy input–correct output pairs.
• Fine-tuned four state-of-the-art sequence-to-sequence models—MASS, UniLM, RoBERTa, and RoBERTa-UniLM—on the synthetic APR dataset for end-to-end readability improvement.
• Adapted RoBERTa for generative tasks using a modified UniLM-style pre-training objective, creating RoBERTa-UniLM to better align with the APR generation task.
• Evaluated models using a hybrid metric suite: readability-aware WER (RA-WER), BLEU, MaxMatch (M2), and GLEU, with RA-WER explicitly designed to penalize errors affecting readability.
• Used a two-stage data pipeline: GEC → TTS → ASR to simulate real-world ASR errors while preserving semantic meaning.
• Compared all models against a traditional rule-based post-processing baseline to isolate the gains from end-to-end learning.

Experimental results

Research questions

• RQ1 Can end-to-end neural models significantly improve the readability of ASR transcripts beyond traditional rule-based post-processing methods?
• RQ2 How effective is synthetic data generation using GEC, TTS, and ASR in creating a high-quality, task-specific dataset for APR?
• RQ3 To what extent do pre-trained models like RoBERTa and RoBERTa-UniLM outperform other architectures (e.g., MASS, UniLM) on the APR task?
• RQ4 Which evaluation metrics—WER, BLEU, M2, GLEU—best reflect human-perceived readability in ASR post-processing?
• RQ5 Does joint error correction and fluency enhancement in a single model lead to better performance than isolated correction steps?

Key findings

• The RoBERTa-based model achieved the best overall performance, with a readability-aware WER (RA-WER) of 13.88 on the JFLEG test set, significantly outperforming the baseline and other models.
• RoBERTa-UniLM showed strong performance on the M2 metric (54.32) and BLEU (80.77), indicating improved fluency and alignment, despite limited training data and convergence.
• The fine-tuned models reduced RA-WER by over 30% compared to the traditional pipeline, demonstrating the effectiveness of end-to-end learning for APR.
• The GLEU metric revealed a trade-off: lower WER and higher BLEU did not always correlate with higher fluency, suggesting that current metrics may not fully capture readability.
• Synthetic data generation via TTS and ASR proved effective in creating a large-scale, diverse APR dataset, enabling strong model generalization.
• Even with limited training steps (7200), RoBERTa-UniLM showed promise, indicating that further pre-training could unlock greater gains.