[Paper Review] Hearing voices at the National Library -- a speech corpus and acoustic model for the Swedish language

Martin Malmsten, Chris Haffenden+1 more|arXiv (Cornell University)|May 6, 2022

Music and Audio Processing5 citations

TL;DR

This paper presents VoxRex, a new monolingual acoustic model for Swedish automatic speech recognition (ASR), trained on a diverse speech corpus—P4—derived from the National Library of Sweden’s audiovisual collections. Using wav2vec 2.0 with minimal preprocessing, VoxRex outperforms existing monolingual and multilingual models, demonstrating the potential of cultural heritage institutions to democratize AI through high-quality, language-specific, unlabelled data.

ABSTRACT

This paper explains our work in developing new acoustic models for automated speech recognition (ASR) at KBLab, the infrastructure for data-driven research at the National Library of Sweden (KB). We evaluate different approaches for a viable speech-to-text pipeline for audiovisual resources in Swedish, using the wav2vec 2.0 architecture in combination with speech corpuses created from KB's collections. These approaches include pretraining an acoustic model for Swedish from the ground up, and fine-tuning existing monolingual and multilingual models. The collections-based corpuses we use have been sampled from millions of hours of speech, with a conscious attempt to balance regional dialects to produce a more representative, and thus more democratic, model. The acoustic model this enabled, "VoxRex", outperforms existing models for Swedish ASR. We also evaluate combining this model with various pretrained language models, which further enhanced performance. We conclude by highlighting the potential of such technology for cultural heritage institutions with vast collections of previously unlabelled audiovisual data. Our models are released for further exploration and research here: https://huggingface.co/KBLab.

Motivation & Objective

• To develop a high-performing, monolingual acoustic model for Swedish ASR using unlabelled audio from the National Library of Sweden’s collections.
• To evaluate the effectiveness of pretraining from scratch versus fine-tuning existing multilingual and monolingual models on Swedish speech data.
• To demonstrate that cultural heritage institutions can play a central role in democratizing AI by leveraging large-scale, unlabelled audio archives for training state-of-the-art ASR systems.
• To explore the potential of unsupervised and self-supervised learning in enabling ASR for lower-resourced languages like Swedish without relying on commercial data pipelines.
• To release a freely available, high-quality ASR model and corpus to support further research and downstream applications in Swedish NLP and audio analysis.

Proposed method

• Constructed the P4 speech corpus by sampling unlabelled audio from local public radio (P4), podcasts, and audiobooks from the National Library of Sweden’s archives, emphasizing regional dialect diversity.
• Trained a new monolingual acoustic model, VoxRex, using the wav2vec 2.0 architecture on the P4 corpus, with minimal preprocessing to detect speech segments.
• Evaluated VoxRex against existing models, including monolingual VoxPopuli-sv and multilingual XLSR-53, using standard ASR benchmarks for Swedish.
• Enhanced performance by combining VoxRex with various pretrained language models, improving automatic speech recognition accuracy.
• Leveraged open-source tools from Facebook AI and Hugging Face for model training, fine-tuning, and deployment, ensuring reproducibility and accessibility.
• Used a CC0 license to release VoxRex and its fine-tuned variants on Hugging Face, enabling broad reuse and further research.

Experimental results

Research questions

• RQ1 Can a high-performing, monolingual ASR model for Swedish be trained from scratch using only unlabelled audio from national cultural heritage collections?
• RQ2 How does a model trained on a diverse, dialect-balanced corpus from public radio and other sources compare to existing multilingual and monolingual models in terms of ASR accuracy?
• RQ3 To what extent can unsupervised pretraining on large-scale, unlabelled audio from national archives reduce reliance on commercial or high-resource data sources?
• RQ4 What is the potential of combining a self-supervised acoustic model with pretrained language models to further improve Swedish speech recognition performance?
• RQ5 How can cultural heritage institutions with vast unlabelled audio collections become active contributors to national AI infrastructure development?

Key findings

• VoxRex, the monolingual acoustic model trained on the P4 corpus, outperforms existing models, including the monolingual VoxPopuli-sv and the multilingual XLSR-53, on Swedish ASR benchmarks.
• The combination of VoxRex with pretrained language models further enhanced ASR performance, indicating strong synergy between acoustic and language modeling components.
• The P4 corpus, derived from local public radio and other sources, provides broad dialectal and speaker diversity, contributing to a more representative and democratic model.
• Despite minimal preprocessing—only basic speech activity detection—VoxRex achieved state-of-the-art performance, indicating the robustness of the wav2vec 2.0 architecture on diverse, real-world audio.
• The model’s success demonstrates that national libraries and cultural heritage institutions can serve as key drivers of AI innovation for lower-resourced languages by leveraging existing, unlabelled audio collections.
• The release of VoxRex under a CC0 license enables widespread reuse and fine-tuning, supporting downstream research and applications in Swedish NLP and audio analysis.