[Paper Review] A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding

Yingzhi Wang, Abdelmoumene Boumadane+1 more|arXiv (Cornell University)|Nov 4, 2021

Speech Recognition and Synthesis27 references93 citations

TL;DR

The paper benchmarks fine-tuned wav2vec 2.0 and HuBERT models on three non-ASR tasks (SER, SV, SLU), achieving state-of-the-art results across datasets with partial and full fine-tuning strategies.

ABSTRACT

Speech self-supervised models such as wav2vec 2.0 and HuBERT are making revolutionary progress in Automatic Speech Recognition (ASR). However, they have not been totally proven to produce better performance on tasks other than ASR. In this work, we explored partial fine-tuning and entire fine-tuning on wav2vec 2.0 and HuBERT pre-trained models for three non-ASR speech tasks: Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding. With simple proposed downstream frameworks, the best scores reached 79.58% weighted accuracy on speaker-dependent setting and 73.01% weighted accuracy on speaker-independent setting for Speech Emotion Recognition on IEMOCAP, 2.36% equal error rate for Speaker Verification on VoxCeleb1, 89.38% accuracy for Intent Classification and 78.92% F1 for Slot Filling on SLURP, showing the strength of fine-tuned wav2vec 2.0 and HuBERT on learning prosodic, voice-print and semantic representations.

Motivation & Objective

• Demonstrate the effectiveness of fine-tuned wav2vec 2.0 and HuBERT on non-ASR speech tasks.
• Compare partial fine-tuning versus entire fine-tuning for downstream tasks.
• Provide open-source fine-tuned models and downstream frameworks for SER and SLU.

Proposed method

• Compare four pre-trained models (wav2vec 2.0 base/large with and without ASR fine-tuning; HuBERT base/large with and without ASR fine-tuning).
• Fine-tune by freezing CNN encoder (partial) or fine-tuning all layers (entire).
• Attach simple downstream adaptors: average pooling + linear classifier for SER and SV; attentional decoder for SLU.
• Train with Adam with separate schedulers for encoder and downstream components.
• Evaluate on IEMOCAP (SER), VoxCeleb1 (SV), and SLURP (SLU).

Experimental results

Research questions

• RQ1 Can fine-tuning self-supervised encoders improve SER, SV, and SLU beyond frozen-feature baselines?
• RQ2 Is partial fine-tuning more effective than full fine-tuning for data-limited SER?
• RQ3 Does HuBERT outperform wav2vec 2.0 on these downstream tasks?
• RQ4 What is the impact of ASR fine-tuning on non-ASR task performance?
• RQ5 How do different model scales (base vs large) affect results across tasks?

Key findings

• Partial fine-tuned HuBERT large achieves 79.58% WA on SER-SD (best) and 73.01% WA on SER-SI.
• Entirely fine-tuned HuBERT large with ASR fine-tuning reaches 2.36% EER on SV (best) and 2.38% EER in PF-960h variant.
• HuBERT generally outperforms wav2vec 2.0 for SER and SV across settings.
• SLU results show IC 89.38% and SF 78.92% with EF-hbt-large and PF-hbt-large, respectively.
• ASR fine-tuning does not consistently help SER and SLU, suggesting loss of prosodic/semantic information in some cases.