[Paper Review] Scribosermo: Fast Speech-to-Text models for German and other Languages

Daniel Bermuth, Alexander Poeppel+1 more|arXiv (Cornell University)|Oct 15, 2021

Speech Recognition and Synthesis19 references4 citations

TL;DR

This paper introduces Scribosermo, a family of small, efficient speech-to-text models for German, Spanish, and French that achieve state-of-the-art performance on German benchmarks while running in real-time on low-power hardware like Raspberry Pi. The models leverage transfer learning from a pretrained English model and a novel alphabet-adaptation technique, enabling high accuracy with minimal data and consumer-grade training hardware.

ABSTRACT

Recent Speech-to-Text models often require a large amount of hardware resources and are mostly trained in English. This paper presents Speech-to-Text models for German, as well as for Spanish and French with special features: (a) They are small and run in real-time on microcontrollers like a RaspberryPi. (b) Using a pretrained English model, they can be trained on consumer-grade hardware with a relatively small dataset. (c) The models are competitive with other solutions and outperform them in German. In this respect, the models combine advantages of other approaches, which only include a subset of the presented features. Furthermore, the paper provides a new library for handling datasets, which is focused on easy extension with additional datasets and shows an optimized way for transfer-learning new languages using a pretrained model from another language with a similar alphabet.

Motivation & Objective

• To develop small, fast speech-to-text models for German and other languages that can run efficiently on embedded systems like Raspberry Pi.
• To enable training of these models on consumer-grade hardware using relatively small datasets, reducing reliance on high-end server resources.
• To improve performance on German speech recognition, achieving state-of-the-art results on the Tuda dataset.
• To provide a streamlined, extensible framework for dataset handling and multilingual transfer learning across languages with similar alphabets.
• To demonstrate effective zero-shot or few-shot transfer learning from English to other Romance languages using a single-step adaptation process.

Proposed method

• The models are based on a lightweight neural network architecture, optimized for inference speed and low memory usage, enabling real-time inference on microcontrollers.
• A pretrained English speech-to-text model is fine-tuned using a novel alphabet-adaptation technique that reconfigures the output layer for target language character sets in a single training step.
• A new open-source library, corcua, is introduced to streamline dataset loading, preprocessing, and conversion across multiple speech recognition frameworks.
• Data preprocessing includes automatic cleaning of audio and transcription files based on duration, transcription length, and speech rate thresholds to improve training stability and speed.
• Language model rescoring with a 5-gram model is applied post-decoding to improve transcription quality, especially for German.
• Transfer learning is applied across languages with similar alphabets (e.g., English → Spanish, Spanish → Italian), using a shared encoder and adapting only the output layer.

Experimental results

Research questions

• RQ1 Can small, efficient ASR models achieve competitive performance on German, Spanish, and French benchmarks while running in real-time on low-power devices?
• RQ2 Can transfer learning from a pretrained English model significantly reduce data and hardware requirements for training new language models?
• RQ3 Does a single-step alphabet-adaptation technique outperform traditional fine-tuning approaches in terms of speed and performance for new languages?
• RQ4 Can a unified, extensible dataset processing pipeline like corcua simplify the integration of diverse multilingual datasets for ASR training?
• RQ5 To what extent does combining multiple small datasets improve model performance without requiring large-scale data or high-end training infrastructure?

Key findings

• The Scribosermo model achieves a Word Error Rate (WER) of 6.6% on the German CommonVoice dataset, outperforming previous models and setting a new state-of-the-art for German speech recognition.
• On the German Tuda dataset, the model achieves a WER of 10.2%, which is competitive with the best existing models, including those from IMS-Speech.
• The model runs in real-time on a Raspberry Pi, demonstrating its suitability for edge and embedded deployment with low power consumption.
• Training the model on just 2.4k hours of German data (from multiple sources) achieves strong performance, indicating high data efficiency.
• The transfer learning approach enables effective adaptation from English to Spanish (WER: 10.0%) and French (WER: 11.0%) using only 817 and 1028 hours of data, respectively.
• The corcua library reduces dataset conversion time from 12 hours (with audiomate) to under 3 hours for a 777-hour dataset, significantly improving data pipeline efficiency.