[Paper Review] AcoustEmo: Open-Vocabulary Emotion Reasoning via Utterance-Aware Acoustic Q-Former

Liyun Zhang, Xuanmeng Sha+2 more|arXiv (Cornell University)|Mar 21, 2026

Emotion and Mood Recognition0 citations

TL;DR

AcoustEmo introduces an utterance-aware acoustic Q-Former with a timestamp-synchronized sliding window to extract segment-level audio tokens, enabling fine-grained open-vocabulary emotion reasoning in MLLMs and outperforming baselines on EMER-Fine.

ABSTRACT

Multimodal Large Language Models (MLLMs) excel in Open-Vocabulary (OV) emotion recognition but often neglect fine-grained acoustic modeling. Existing methods typically use global audio encoders, failing to capture subtle, local temporal dynamics like micro-prosody and intonation shifts within individual utterances. To address this, we propose AcoustEmo, a time-sensitive MLLM featuring a novel Utterance-Aware Acoustic Q-Former. Our approach utilizes a timestamp-synchronized sliding window to dynamically extract segment-level audio tokens instead of coarse global representations. This enables the model to explicitly trace the temporal evolution of subtle acoustic clues and capture deep contextual dependencies in dialogues. Experiments on the Explainable Multimodal Emotion Recognition (EMER) task show that AcoustEmo significantly enhances complex emotion reasoning, outperforming baselines while maintaining robust contextual accuracy.

Motivation & Objective

• Address limitations of global audio encoders that miss micro-prosody and local temporal dynamics in emotion reasoning.
• Develop an utterance-aware acoustic Q-Former to extract segment-level tokens aligned with transcription timestamps.
• Integrate fine-grained acoustic tokens with visual and textual modalities for open-vocabulary emotion reasoning in MLLMs.
• Evaluate on EMER task to demonstrate improvements over state-of-the-art baselines.

Proposed method

• Replace global audio encoder with Utterance-Aware Acoustic Q-Former that processes per-utterance acoustic segments.
• Use a timestamp-synchronized sliding window to extract F_A^i for each utterance i from the frame-level audio features.
• Employ learnable queries Q in cross-attention to distill salient micro-prosodic details for each utterance.
• Combine global acoustic tokens with utterance-specific tokens into T_A and concatenate with visual tokens T_V and instruction tokens T_Lq for LLM reasoning.
• Fine-tune with LoRA on an LLaMA-2 backbone while freezing visual/acoustic encoders.

Experimental results

Research questions

• RQ1 Can utterance-level, timestamp-aligned acoustic representations improve open-vocabulary emotion reasoning in MLLMs?
• RQ2 Do local acoustic dynamics (micro-prosody) contribute more to emotion reasoning than global audio pooling?
• RQ3 How does timestamp synchronization affect the alignment between acoustic cues and textual utterances in EMER tasks?

Key findings

• AcoustEmo achieves an Avg score of 67.55 with Accuracy 65.40 and Recall 70.15 on EMER-Fine.
• AcoustEmo outperforms many baselines, including AffectGPT and MicroEmo, and approaches or surpasses several video-centric and audio-centric models depending on the setup.
• Ablation shows removing Utterance-Aware Acoustic Q-Former drops Avg to 61.20, highlighting the importance of local acoustic dynamics.
• Replacing the timestamp-synchronized window with a fixed 2s window lowers Avg to 62.85, demonstrating the need for precise utterance alignment.
• Removing the Global Acoustic Q-Former reduces Avg to 64.10, indicating global context still provides useful background information.