[Paper Review] Cross-Language Speech Emotion Recognition Using Multimodal Dual Attention Transformers

Syed Aun Muhammad Zaidi, Siddique Latif+1 more|arXiv (Cornell University)|Jun 23, 2023

Emotion and Mood RecognitionPsychology8 citations

TL;DR

MDAT leverages multilingual pre-trained models for speech and text, using dual attention (graph attention and co-attention) and a transformer encoder to improve cross-language SER with limited target-language data.

ABSTRACT

Despite the recent progress in speech emotion recognition (SER), state-of-the-art systems are unable to achieve improved performance in cross-language settings. In this paper, we propose a Multimodal Dual Attention Transformer (MDAT) model to improve cross-language SER. Our model utilises pre-trained models for multimodal feature extraction and is equipped with a dual attention mechanism including graph attention and co-attention to capture complex dependencies across different modalities and achieve improved cross-language SER results using minimal target language data. In addition, our model also exploits a transformer encoder layer for high-level feature representation to improve emotion classification accuracy. In this way, MDAT performs refinement of feature representation at various stages and provides emotional salient features to the classification layer. This novel approach also ensures the preservation of modality-specific emotional information while enhancing cross-modality and cross-language interactions. We assess our model's performance on four publicly available SER datasets and establish its superior effectiveness compared to recent approaches and baseline models.

Motivation & Objective

• Address cross-language SER with limited target-language data.
• Leverage multilingual pre-trained models for speech and text features.
• Preserve modality-specific emotional information while enhancing cross-modality interactions.
• Employ dual attention and transformer encoder to improve emotion classification across languages.

Proposed method

• Use XLS-R (multilingual wav2vec 2.0) for speech embeddings and RoBERTa for text embeddings.
• Apply a 1x1 conv to align text feature dimension to speech features.
• Employ a graph attention layer to model intra-modality dependencies.
• Apply a co-attention layer to capture cross-modality interactions between speech and text.
• Concatenate inputs with attended features and pass through a transformer encoder for high-level representation.
• Use a dense-softmax classifier on concatenated transformer outputs.

Experimental results

Research questions

• RQ1 Can a multimodal transformer with dual attention improve cross-language SER with limited target-language data?
• RQ2 Do multilingual pre-trained models for speech and text enhance generalisation across languages?
• RQ3 Does combining graph attention and co-attention preserve modality-specific info while improving cross-modality fusion?
• RQ4 How does MDAT compare to baseline BiLSTM and other multimodal SER methods across languages?

Key findings

• MDAT outperforms baselines and prior multimodal methods across four language datasets in within-corpus evaluations.
• On IEMOCAP, URDU, EMODB, and EMOVO, MDAT achieves higher unweighted accuracy than Baseline, SAFRLM, and HCAM.
• The model demonstrates superior cross-language adaptability by leveraging cross-language pre-trained representations.
• Transformer-based high-level representation learning further improves emotion classification accuracy over dual-attention-only approaches.
• MDAT achieves the highest reported UA in the provided within-corpus comparisons.