[Paper Review] Leveraging neural representations for facilitating access to untranscribed speech from endangered languages.

Nay San, Martijn Bartelds+10 more|arXiv (Cornell University)|Mar 26, 2021

Speech Recognition and Synthesis8 references4 citations

TL;DR

This paper proposes leveraging pre-trained English wav2vec 2.0 representations to improve query-by-example spoken term detection (QbE-STD) in low-resource, endangered languages. By extracting intermediate-layer features from the pre-trained model, the method achieves 56–86% relative performance gains over traditional features like Mel-frequency cepstral coefficients and bottleneck features, with greater gains for languages phonologically closer to English.

ABSTRACT

For languages with insufficient resources to train speech recognition systems, query-by-example spoken term detection (QbE-STD) offers a way of accessing an untranscribed speech corpus by helping identify regions where spoken query terms occur. Yet retrieval performance can be poor when the query and corpus are spoken by different speakers and produced in different recording conditions. Using data selected from a variety of speakers and recording conditions from 7 Australian Aboriginal languages and a regional variety of Dutch, all of which are endangered or vulnerable, we evaluated whether QbE-STD performance on these languages could be improved by leveraging representations extracted from the pre-trained English wav2vec 2.0 model. Compared to the use of Mel-frequency cepstral coefficients and bottleneck features, we find that representations from the middle layers of the wav2vec 2.0 Transformer offer large gains in task performance (between 56% and 86%). While features extracted using the pre-trained English model yielded improved detection on all the evaluation languages, better detection performance was associated with the evaluation language's phonological similarity to English.

Motivation & Objective

• To address poor retrieval performance in query-by-example spoken term detection (QbE-STD) for low-resource, endangered languages due to speaker and recording condition variability.
• To investigate whether pre-trained neural representations from a high-resource language (English) can enhance QbE-STD performance in low-resource, endangered languages.
• To evaluate the impact of phonological similarity between the target language and English on the effectiveness of transferred representations.
• To compare the performance of pre-trained wav2vec 2.0 features against traditional acoustic features like Mel-frequency cepstral coefficients and bottleneck features.

Proposed method

• Fine-tuned and evaluated query-by-example spoken term detection (QbE-STD) systems on untranscribed speech corpora from 7 Australian Aboriginal languages and a regional Dutch variety.
• Extracted intermediate-layer representations from a pre-trained English wav2vec 2.0 model, using the Transformer encoder's hidden states as acoustic features.
• Compared the performance of these representations against standard features such as Mel-frequency cepstral coefficients and bottleneck features.
• Evaluated performance across diverse speakers and recording conditions to assess robustness to domain shift.
• Measured detection performance using standard metrics like equal error rate (EER) and relative improvement over baseline features.
• Analyzed the correlation between phonological similarity to English and performance gains from using pre-trained representations.

Experimental results

Research questions

• RQ1 Can pre-trained English wav2vec 2.0 representations improve QbE-STD performance in low-resource, endangered languages with limited transcribed data?
• RQ2 How does the performance of transferred representations vary across languages with differing phonological similarity to English?
• RQ3 To what extent do intermediate-layer features from a pre-trained model outperform traditional acoustic features like Mel-frequency cepstral coefficients and bottleneck features in QbE-STD?
• RQ4 Does the use of pre-trained representations mitigate performance degradation caused by speaker and recording condition mismatches?

Key findings

• Pre-trained English wav2vec 2.0 representations from middle layers of the Transformer encoder significantly improved QbE-STD performance across all evaluated endangered languages.
• Relative performance gains ranged from 56% to 86% compared to baseline features like Mel-frequency cepstral coefficients and bottleneck features.
• Languages with higher phonological similarity to English showed greater performance improvements when using the pre-trained representations.
• The use of pre-trained features yielded consistent improvements across all evaluation languages, indicating broad transferability despite low-resource conditions.
• The best-performing features were derived from intermediate layers of the wav2vec 2.0 model, not the final layer or raw input.
• The results demonstrate that pre-trained representations can effectively bridge domain gaps in low-resource speech retrieval tasks.