[Paper Review] Introducing ECAPA-TDNN and Wav2Vec2.0 Embeddings to Stuttering Detection

Shakeel A. Sheikh, Md Sahidullah+2 more|arXiv (Cornell University)|Apr 4, 2022

Stuttering Research and Treatment4 citations

TL;DR

This paper introduces ECAPA-TDNN and Wav2Vec2.0 embeddings for stuttering detection, leveraging pre-trained models on large-scale audio datasets to overcome limited stuttering data. It achieves a 16.74% relative accuracy improvement over baseline by fusing ECAPA-TDNN speaker embeddings with Wav2Vec2.0 contextual representations, especially through multi-layer concatenation and score fusion.

ABSTRACT

The adoption of advanced deep learning (DL) architecture in stuttering detection (SD) tasks is challenging due to the limited size of the available datasets. To this end, this work introduces the application of speech embeddings extracted with pre-trained deep models trained on massive audio datasets for different tasks. In particular, we explore audio representations obtained using emphasized channel attention, propagation, and aggregation-time-delay neural network (ECAPA-TDNN) and Wav2Vec2.0 model trained on VoxCeleb and LibriSpeech datasets respectively. After extracting the embeddings, we benchmark with several traditional classifiers, such as a k-nearest neighbor, Gaussian naive Bayes, and neural network, for the stuttering detection tasks. In comparison to the standard SD system trained only on the limited SEP-28k dataset, we obtain a relative improvement of 16.74% in terms of overall accuracy over baseline. Finally, we have shown that combining two embeddings and concatenating multiple layers of Wav2Vec2.0 can further improve SD performance up to 1% and 2.64% respectively.

Motivation & Objective

• Address the challenge of limited, biased stuttering datasets in training deep learning models for stuttering detection.
• Explore the transferability of pre-trained speech embeddings—specifically ECAPA-TDNN and Wav2Vec2.0—for stuttering detection.
• Investigate the impact of combining ECAPA-TDNN and Wav2Vec2.0 embeddings via score and embedding fusion to improve detection performance.
• Analyze the contribution of different Wav2Vec2.0 hidden layers and their concatenation for capturing disfluent speech patterns.
• Demonstrate that contextual and speaker-level representations from pre-trained models significantly enhance detection of rare stuttering types.

Proposed method

• Extract 192-dimensional speaker embeddings from the fully connected layer of pre-trained ECAPA-TDNN, fine-tuned on VoxCeleb for speaker verification.
• Obtain contextual speech representations from multiple layers (L1, L7, L11) of the Wav2Vec2.0 model pre-trained on LibriSpeech, using the local encoder and contextual block.
• Apply Linear Discriminant Analysis (LDA) to reduce dimensionality of Wav2Vec2.0 embeddings to improve class separability in downstream classification.
• Train and evaluate multiple classifiers—k-Nearest Neighbors (KNN), Naïve Bayes (NBC), and Multilayer Perceptron (MLP)—on the extracted embeddings.
• Implement score fusion by combining prediction probabilities from ECAPA-TDNN and Wav2Vec2.0 models using an optimized weighting parameter α=0.9.
• Perform embedding fusion by concatenating ECAPA-TDNN and Wav2Vec2.0 embeddings before feeding into classifiers to leverage complementary information.

Experimental results

Research questions

• RQ1 Can pre-trained ECAPA-TDNN and Wav2Vec2.0 embeddings improve stuttering detection performance on small datasets like SEP-28k?
• RQ2 How does the choice of Wav2Vec2.0 layer (L1, L7, L11) affect detection performance, and does concatenating multiple layers further improve results?
• RQ3 To what extent does score fusion between ECAPA-TDNN and Wav2Vec2.0 embeddings enhance detection accuracy compared to individual models?
• RQ4 Does combining ECAPA-TDNN and Wav2Vec2.0 embeddings through embedding-level fusion yield better performance than score fusion?
• RQ5 How do the embeddings perform across different stuttering types (blocks, repetitions, prolongations, interjections, fluents), especially minority classes?

Key findings

• Using Wav2Vec2.0 embeddings with a neural network classifier achieves a 16.74% relative improvement in overall accuracy over the baseline system trained only on SEP-28k.
• Fusing ECAPA-TDNN and Wav2Vec2.0 embeddings via score fusion improves overall accuracy to 67.26% with a neural network, outperforming individual models.
• Concatenating representations from L1, L7, and L11 layers of Wav2Vec2.0 boosts detection performance by 2.64% over the baseline, especially improving recognition of minority classes like blocks and prolongations.
• The use of LDA on Wav2Vec2.0 embeddings increases detection performance by 7.28% overall, with the largest gains in blocks (49.88%) and prolongations (14.77%).
• ECAPA-TDNN embeddings enhance detection of fluent speech by 5.34% (KNN), 1.13% (NBC), and 5.13% (NN) over Wav2Vec2.0 alone, indicating strong speaker identity information.
• Embedding fusion of ECAPA-TDNN and Wav2Vec2.0 achieves the highest overall accuracy of 68.35% with a neural network, with significant gains in blocks (23.86%) and interjections (69.54%) compared to baseline.