[Paper Review] How Attention Shapes Emotion: A Comparative Study of Attention Mechanisms for Speech Emotion Recognition

Marc Casals-Salvador, Federico Costa+2 more|arXiv (Cornell University)|Mar 16, 2026

Emotion and Mood Recognition0 citations

TL;DR

This paper benchmarks efficient attention mechanisms (RetNet, LightNet, GSA, FoX, KDA) against standard self-attention for Speech Emotion Recognition (SER) on MSP-Podcast, evaluating accuracy and efficiency trade-offs.

ABSTRACT

Speech Emotion Recognition (SER) plays a key role in advancing human-computer interaction. Attention mechanisms have become the dominant approach for modeling emotional speech due to their ability to capture long-range dependencies and emphasize salient information. However, standard self-attention suffers from quadratic computational and memory complexity, limiting its scalability. In this work, we present a systematic benchmark of optimized attention mechanisms for SER, including RetNet, LightNet, GSA, FoX, and KDA. Experiments on both MSP-Podcast benchmark versions show that while standard self-attention achieves the strongest recognition performance across test sets, efficient attention variants dramatically improve scalability, reducing inference latency and memory usage by up to an order of magnitude. These results highlight a critical trade-off between accuracy and efficiency, providing practical insights for designing scalable SER systems.

Motivation & Objective

• Assess how different seq2seq attention mechanisms affect SER performance and scalability.
• Benchmark state-of-the-art efficient attention variants against standard self-attention under unified settings.
• Analyze memory, training/inference time, and robustness across MSP-Podcast versions.

Proposed method

• Process speech and text through fixed feature extractors and fuse via a seq2seq module with different attention mechanisms.
• Compare Softmax Attention (SA) against RetNet, LightNet, GSA, FoX, and KDA under identical architectures and datasets.
• Measure Macro F-score on Dev, Test1 (T1), and Test2 (T2) across multiple SSL backbones.
• Evaluate inference latency and peak GPU memory to quantify efficiency.
• Freeze feature extractors; train only seq2seq, attention pooling, and classifier.
• Use 20 epochs, AdamW optimization, and 1x batch size for evaluation of efficiency results.

Figure 1: System's architecture. Experiments are made considering different attention mechanisms for the seq2seq module.

Experimental results

Research questions

• RQ1 Does each attention mechanism provide the best trade-off between SER accuracy and computational efficiency on MSP-Podcast?
• RQ2 How do efficient attention variants compare to SA in inference latency and memory usage as sequence length grows?
• RQ3 What is the robustness of each mechanism across MSP-Podcast Test1 and Test2 conditions?
• RQ4 How does the choice of SSL backbone influence the relative performance of each attention mechanism?

Key findings

• Softmax Attention (SA) yields the strongest generalization on evaluation splits (Test1 and Test2) overall, despite efficient variants offering better scalability.
• LightNet achieves the highest mean development score (Dev) overall, with a strongest single result of 38.11% using Wav2Vec2XLSR.
• Efficient mechanisms scale linearly in inference time and memory with sequence length, while SA scales quadratically (0.55 ms at 10 s vs. 48.59 ms at 400 s).
• KDA is the fastest among efficient mechanisms at long sequences (5.96 ms at 400 s), FoX is the most memory-efficient (0.328 GB at 400 s).
• All methods see performance drops from Test1 to Test2, indicating robustness gaps under more realistic and imbalanced conditions.
• Efficient architectures can approach SA’s accuracy while offering substantial improvements in latency and memory usage.

Figure 2: Inference time and peak GPU memory usage of the seq2seq module as a function of sequence length on the MSP-Podcast dev set [ 8003425 ] . Panels (a–b) report results for all models. Panels (c–d) provide a zoomed view excluding SA to make the relative growth trends of the remaining alternati