[Paper Review] Transfer learning from High-Resource to Low-Resource Language Improves Speech Affect Recognition Classification Accuracy

Sara Durrani, Muhammad Umair Arshad|arXiv (Cornell University)|Mar 4, 2021

Emotion and Mood Recognition4 citations

TL;DR

This paper proposes a transfer learning approach that leverages high-resource language data (IEMOCAP, EMOVO) to improve speech affect recognition in low-resource languages (Urdu, Italian), achieving UAR scores of 69.32% and 68.52% respectively. By fine-tuning a pre-trained encoder-decoder model with attention on target language data, the method significantly boosts performance over baseline and cross-corpus settings, demonstrating that transfer of both encoder and attention parameters is essential for success.

ABSTRACT

Speech Affect Recognition is a problem of extracting emotional affects from audio data. Low resource languages corpora are rear and affect recognition is a difficult task in cross-corpus settings. We present an approach in which the model is trained on high resource language and fine-tune to recognize affects in low resource language. We train the model in same corpus setting on SAVEE, EMOVO, Urdu, and IEMOCAP by achieving baseline accuracy of 60.45, 68.05, 80.34, and 56.58 percent respectively. For capturing the diversity of affects in languages cross-corpus evaluations are discussed in detail. We find that accuracy improves by adding the domain target data into the training data. Finally, we show that performance is improved for low resource language speech affect recognition by achieving the UAR OF 69.32 and 68.2 for Urdu and Italian speech affects.

Motivation & Objective

• To address the challenge of limited annotated data in low-resource languages for speech affect recognition.
• To investigate whether transfer learning from high-resource to low-resource languages improves classification accuracy.
• To evaluate the effectiveness of joint training on multiple corpora for cross-lingual affect recognition.
• To determine which model components (encoder, decoder, attention) are most critical for successful transfer.

Proposed method

• A sequence-to-sequence encoder-decoder model with attention mechanism is used for end-to-end speech affect recognition.
• The model is pre-trained on high-resource corpora (IEMOCAP and EMOVO) to learn general affective patterns.
• Low-resource language data (Urdu and Italian) is fine-tuned using the pre-trained model, with domain-specific data added during training.
• Transfer learning is applied by initializing the target language model with encoder and attention weights from the source language model.
• Three-fold cross-validation is used to evaluate generalization across corpora.
• Unweighted average recall (UAR) is used as the primary metric to handle class imbalance in affective speech datasets.

Experimental results

Research questions

• RQ1 Can transfer learning from high-resource to low-resource languages improve speech affect recognition accuracy?
• RQ2 How does joint training on multiple corpora affect model generalization across different languages and affective classes?
• RQ3 Which components of the pre-trained model (encoder, decoder, attention) are most effective for transfer in low-resource settings?
• RQ4 Does adding domain-specific target language data during fine-tuning enhance performance?

Key findings

• The transfer learning approach achieved a UAR of 69.32% on Urdu and 68.52% on Italian, significantly outperforming baseline and cross-corpus settings.
• The model trained on IEMOCAP and fine-tuned on Urdu achieved a UAR of 69.32% on Urdu, demonstrating effective knowledge transfer to low-resource languages.
• Adding domain-specific target language data during training improved classification accuracy, highlighting the importance of in-domain fine-tuning.
• Transferring both encoder and attention parameters was crucial for performance gains; transferring only decoder parameters yielded no significant improvement.
• The model outperformed existing approaches in cross-corpus evaluations, especially when trained on high-resource corpora and tested on low-resource ones.
• The results show that low-resource languages like Urdu and Italian achieved higher accuracy than high-resource English in some transfer settings, indicating the potential of transfer learning to mitigate data scarcity.