[Paper Review] A Transfer Learning Method for Speech Emotion Recognition from Automatic Speech Recognition

Zhou Sitong, Homayoon Beigi|arXiv (Cornell University)|Aug 6, 2020

Emotion and Mood RecognitionPsychology20 references17 citations

TL;DR

This paper proposes a transfer learning method for speech emotion recognition by fine-tuning a TDNN model pre-trained on the large-scale Tedlium2 ASR dataset, using speaker-adapted MFCC and i-vector features. The approach achieves 71.7% unweighted accuracy on the IEMOCAP dataset, significantly outperforming prior state-of-the-art methods.

ABSTRACT

This paper presents a transfer learning method in speech emotion recognition based on a Time-Delay Neural Network (TDNN) architecture. A major challenge in the current speech-based emotion detection research is data scarcity. The proposed method resolves this problem by applying transfer learning techniques in order to leverage data from the automatic speech recognition (ASR) task for which ample data is available. Our experiments also show the advantage of speaker-class adaptation modeling techniques by adopting identity-vector (i-vector) based features in addition to standard Mel-Frequency Cepstral Coefficient (MFCC) features.[1] We show the transfer learning models significantly outperform the other methods without pretraining on ASR. The experiments performed on the publicly available IEMOCAP dataset which provides 12 hours of motional speech data. The transfer learning was initialized by using the Ted-Lium v.2 speech dataset providing 207 hours of audio with the corresponding transcripts. We achieve the highest significantly higher accuracy when compared to state-of-the-art, using five-fold cross validation. Using only speech, we obtain an accuracy 71.7% for anger, excitement, sadness, and neutrality emotion content.

Motivation & Objective

• To address data scarcity in speech emotion recognition by leveraging large-scale ASR datasets as a pretraining source.
• To improve emotion recognition performance through transfer learning from automatic speech recognition (ASR) to emotion classification.
• To enhance model robustness by integrating i-vector-based speaker adaptation with MFCC features.
• To evaluate the effectiveness of transfer learning using a 5-fold cross-validation strategy on the IEMOCAP benchmark.

Proposed method

• A TDNN-based deep neural network is pre-trained on 207 hours of Tedlium2 ASR data to learn general acoustic representations.
• Speaker-adapted features are created by concatenating 40-dimensional MFCCs with 100-dimensional i-vectors at each frame.
• The 12th layer of the pre-trained TDNN is selected as the bottleneck feature layer for fine-tuning on the IEMOCAP emotion dataset.
• Fine-tuning is performed using a fully connected layer and softmax classifier for frame-level emotion classification.
• Utterance-level predictions are obtained by aggregating frame-level predictions using max-likelihood voting.
• A 5-fold cross-validation strategy is used, with each session of IEMOCAP used once as the test set.

Experimental results

Research questions

• RQ1 Can pre-training on large-scale ASR data improve performance in low-resource speech emotion recognition?
• RQ2 How does the integration of i-vector speaker adaptation affect emotion recognition accuracy?
• RQ3 Which layer in a pre-trained TDNN architecture yields the best transferable features for emotion classification?
• RQ4 Does transfer learning from ASR lead to significant performance gains over end-to-end training on limited emotion datasets?

Key findings

• The proposed transfer learning method achieves 71.7% unweighted accuracy on the IEMOCAP dataset using 5-fold cross-validation, surpassing the previous state-of-the-art of 68.8%.
• The 12th TDNN layer yields the highest performance (69.3% on session 5), outperforming the prefinal layer (63.4%) and 13th layer (66.7%), indicating its optimal balance of generalization and discriminability.
• The model shows particular difficulty distinguishing excitement from neutrality, with 28% of excitement samples misclassified as neutral.
• Neutrality is the most frequent source of misclassification, often confused with sadness due to low-energy, monotone speech patterns.
• The use of i-vector features significantly improves robustness to speaker and environmental variability in emotion recognition.
• The results demonstrate that high-level acoustic representations learned from ASR transfer effectively to emotion recognition, suggesting shared underlying representations between speech recognition and emotion detection.