[Paper Review] Effectiveness of Mining Audio and Text Pairs from Public Data for Improving ASR Systems for Low-Resource Languages

Kaushal Santosh Bhogale, Abhigyan Raman+5 more|arXiv (Cornell University)|Aug 26, 2022

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes mining high-quality audio-text pairs from public archives of All India Radio to create Shrutilipi, a 6,457-hour multilingual ASR dataset for 12 Indian languages. By adapting the Needleman-Wunsch algorithm to align OCR-transcribed text with CTC-aligned ASR outputs, the method achieves robust alignment despite OCR errors and non-transcribed audio segments, resulting in a 5.8% average WER reduction on IndicSUPERB benchmarks when fine-tuning Wav2Vec models.

ABSTRACT

End-to-end (E2E) models have become the default choice for state-of-the-art speech recognition systems. Such models are trained on large amounts of labelled data, which are often not available for low-resource languages. Techniques such as self-supervised learning and transfer learning hold promise, but have not yet been effective in training accurate models. On the other hand, collecting labelled datasets on a diverse set of domains and speakers is very expensive. In this work, we demonstrate an inexpensive and effective alternative to these approaches by ``mining'' text and audio pairs for Indian languages from public sources, specifically from the public archives of All India Radio. As a key component, we adapt the Needleman-Wunsch algorithm to align sentences with corresponding audio segments given a long audio and a PDF of its transcript, while being robust to errors due to OCR, extraneous text, and non-transcribed speech. We thus create Shrutilipi, a dataset which contains over 6,400 hours of labelled audio across 12 Indian languages totalling to 4.95M sentences. On average, Shrutilipi results in a 2.3x increase over publicly available labelled data. We establish the quality of Shrutilipi with 21 human evaluators across the 12 languages. We also establish the diversity of Shrutilipi in terms of represented regions, speakers, and mentioned named entities. Significantly, we show that adding Shrutilipi to the training set of Wav2Vec models leads to an average decrease in WER of 5.8\% for 7 languages on the IndicSUPERB benchmark. For Hindi, which has the most benchmarks (7), the average WER falls from 18.8% to 13.5%. This improvement extends to efficient models: We show a 2.3% drop in WER for a Conformer model (10x smaller than Wav2Vec). Finally, we demonstrate the diversity of Shrutilipi by showing that the model trained with it is more robust to noisy input.

Motivation & Objective

• To address the lack of large-scale, diverse, and high-quality labelled audio data for low-resource Indian languages in automatic speech recognition (ASR).
• To develop a scalable and robust method for mining audio and text pairs from publicly available, unstructured archives such as All India Radio bulletins.
• To evaluate the effectiveness of the mined dataset in improving ASR performance across diverse benchmarks, including noisy conditions and efficient model architectures.
• To demonstrate that mining public data can significantly close the performance gap between high-resource and low-resource languages in end-to-end ASR systems.

Proposed method

• The method processes long-form audio bulletins from All India Radio using an ASR system (IndicWav2Vec) to generate CTC-aligned character sequences for audio segments.
• Text is extracted from PDF transcripts using OCR, followed by text cleanup to handle formatting, metadata, and non-transcribed content.
• A modified Needleman-Wunsch algorithm is applied to align transcribed text segments with corresponding audio segments, with penalties for insertions, deletions, and mismatches to handle OCR errors and speech irregularities.
• The alignment process is optimized to tolerate non-transcribed speech, background music, and code-mixing by focusing on high-similarity sentence pairs across audio and text.
• The resulting dataset, Shrutilipi, contains over 6,400 hours of audio and 4.95 million sentences across 12 Indian languages, with rigorous quality evaluation by 21 human annotators.
• The effectiveness of Shrutilipi is evaluated by fine-tuning Wav2Vec and Conformer models on multiple benchmarks, including clean and noisy conditions.

Experimental results

Research questions

• RQ1 Can publicly available, unstructured audio and text archives be effectively mined to create high-quality, large-scale training data for low-resource language ASR?
• RQ2 How effective is a modified Needleman-Wunsch algorithm in aligning audio and text segments when faced with OCR errors, non-transcribed speech, and formatting inconsistencies?
• RQ3 To what extent does adding mined data from Shrutilipi improve ASR performance on low-resource Indian languages compared to existing public datasets?
• RQ4 Does training with Shrutilipi enhance model robustness to noisy inputs, especially in real-world deployment scenarios?

Key findings

• Shrutilipi contains 6,457 hours of labelled audio across 12 Indian languages, representing a 2.3× increase in training data on average over existing public datasets.
• Human evaluation confirms the dataset’s high quality, with 21 evaluators across 12 languages rating the data as reliable and representative of diverse regional and linguistic variation.
• Fine-tuning Wav2Vec models on Shrutilipi leads to an average 5.8% relative WER reduction across 7 Indian languages on the IndicSUPERB benchmark, with Hindi showing a drop from 18.8% to 13.5% WER.
• The Conformer model, which is 10× smaller than Wav2Vec, achieves a 2.3% absolute WER reduction when trained on Shrutilipi, demonstrating effectiveness even for efficient architectures.
• On a hard benchmark with added background noise from the ESC dataset, Shrutilipi-trained models show a 5.9% average WER reduction, indicating improved robustness to noisy inputs.
• The dataset’s diversity in speakers, regions, and named entities is validated, with stronger performance on out-of-domain and noisy test sets compared to models trained on smaller, less diverse data.