[Paper Review] Key-Sparse Transformer for Multimodal Speech Emotion Recognition

Weidong Chen, Xiaofeng Xing+3 more|arXiv (Cornell University)|Jun 22, 2021

Emotion and Mood Recognition32 references4 citations

TL;DR

This paper proposes a key-sparse Transformer (KS-Transformer) for multimodal speech emotion recognition that dynamically identifies and focuses on emotion-relevant speech frames and words while suppressing irrelevant, noisy tokens. By applying a learnable sparsity mechanism to attention weights, the model improves performance on IEMOCAP and LSSED, achieving state-of-the-art results with 75.3% unweighted accuracy on IEMOCAP and 55.7% on LSSED, demonstrating superior robustness and efficiency in multimodal emotion recognition.

ABSTRACT

Speech emotion recognition is a challenging research topic that plays a critical role in human-computer interaction. Multimodal inputs further improve the performance as more emotional information is used. However, existing studies learn all the information in the sample while only a small portion of it is about emotion. The redundant information will become noises and limit the system performance. In this paper, a key-sparse Transformer is proposed for efficient emotion recognition by focusing more on emotion related information. The proposed method is evaluated on the IEMOCAP and LSSED. Experimental results show that the proposed method achieves better performance than the state-of-the-art approaches.

Motivation & Objective

• To address the issue of redundant, non-emotional information in audio and text inputs that degrades performance in speech emotion recognition.
• To improve multimodal fusion efficiency by enabling selective attention to emotion-relevant features in both audio and text modalities.
• To develop a sparse attention mechanism that automatically identifies and prioritizes emotionally salient frames and words.
• To achieve state-of-the-art performance on benchmark datasets IEMOCAP and LSSED through improved modality interaction and feature focusing.

Proposed method

• Proposes a key-sparse attention mechanism that computes attention weights and then resets the smallest weights to zero based on a learnable threshold, achieving sparsity in attention heads.
• Introduces a cascaded cross-attention block (CCAB) that enables deep, multi-stage interaction between audio and text modalities after key-sparse processing.
• Uses pre-trained wav2vec and RoBERTa models to extract audio and text embeddings, respectively, for robust initial feature representation.
• Applies a differentiable sparsity strategy where k=0.5 (50% of attention weights reset to zero) is selected as the optimal balance between information retention and noise suppression.
• Employs a multi-head attention mechanism within the KS-Transformer, with sparsity applied per head to maintain attention diversity.
• Integrates the KS-Transformer into a three-module architecture: feature extraction, modality interaction (via CCAB), and deep fusion for final classification.

Experimental results

Research questions

• RQ1 Can a learnable key-sparse attention mechanism improve speech emotion recognition by filtering out non-emotional, noisy tokens in audio and text?
• RQ2 How does the number of cascaded cross-attention blocks (CCABs) affect modality interaction and overall performance in multimodal emotion recognition?
• RQ3 Does the proposed KS-Transformer outperform standard Transformers and existing state-of-the-art models on benchmark datasets like IEMOCAP and LSSED?
• RQ4 What is the optimal level of sparsity (controlled by hyperparameter k) for balancing information retention and noise reduction in emotion recognition?
• RQ5 How does the model perform under class imbalance, particularly on the large-scale LSSED dataset?

Key findings

• The proposed KS-Transformer achieves 75.3% unweighted accuracy on IEMOCAP, outperforming all prior state-of-the-art methods, including CMA (72.8%), STSER (72.1%), and GBAN (70.1%).
• On the LSSED dataset, the model achieves 55.7% unweighted accuracy, the highest among all compared methods, including PyResNet (42.9%) and STSER (51.2%).
• The optimal sparsity level is k=0.5, where 50% of attention weights are reset to zero, balancing performance and noise suppression across both datasets.
• Using three cascaded cross-attention blocks (CCABs) yields the best performance, indicating that multiple rounds of modality interaction are essential for effective fusion.
• Visualization confirms that KS-Transformer reduces attention on non-emotional words (e.g., 'it', 'a', 'look') while increasing focus on emotionally salient terms like 'beautiful' and 'arguing'.
• The model shows improved robustness to overfitting on IEMOCAP and reduced performance degradation on LSSED when k exceeds 0.5, confirming the benefit of sparsity in handling large-scale, imbalanced data.