[Paper Review] A Unified Framework for Emotion Recognition and Sentiment Analysis via Expert-Guided Multimodal Fusion with Large Language Models

Jiaqi Qiao, Xiujuan Xu+2 more|arXiv (Cornell University)|Jan 12, 2026

Emotion and Mood Recognition0 citations

TL;DR

EGMF proposes a unified multimodal framework that fuses text, audio, and visual signals using three specialized expert networks with hierarchical dynamic gating, integrated with LoRA-tuned LLMs to perform both ERC classification and MSA regression, achieving state-of-the-art results across English and Chinese datasets.

ABSTRACT

Multimodal emotion understanding requires effective integration of text, audio, and visual modalities for both discrete emotion recognition and continuous sentiment analysis. We present EGMF, a unified framework combining expert-guided multimodal fusion with large language models. Our approach features three specialized expert networks--a fine-grained local expert for subtle emotional nuances, a semantic correlation expert for cross-modal relationships, and a global context expert for long-range dependencies--adaptively integrated through hierarchical dynamic gating for context-aware feature selection. Enhanced multimodal representations are integrated with LLMs via pseudo token injection and prompt-based conditioning, enabling a single generative framework to handle both classification and regression through natural language generation. We employ LoRA fine-tuning for computational efficiency. Experiments on bilingual benchmarks (MELD, CHERMA, MOSEI, SIMS-V2) demonstrate consistent improvements over state-of-the-art methods, with superior cross-lingual robustness revealing universal patterns in multimodal emotional expressions across English and Chinese. We will release the source code publicly.

Motivation & Objective

• Motivate unified modeling for ERC and MSA across languages.
• Develop an adaptive, expert-guided fusion mechanism for multimodal signals.
• Leverage large language models with prompt-based conditioning for unified generation tasks.
• Improve computational efficiency with LoRA fine-tuning.
• Demonstrate cross-lingual robustness on English and Chinese datasets.

Proposed method

• Introduce three specialized expert networks: a fine-grained local expert, a semantic correlation expert, and a global context expert.
• Apply hierarchical dynamic gating to adaptively weight and fuse expert outputs.
• Use cross-modal attention to create enhanced multimodal representations.
• Inject pseudo tokens and condition LLM generation with prompts for both classification and regression.
• Fine-tune efficiently using LoRA while keeping the base model frozen.

Figure 1: Architecture of the proposed EGMF framework.

Experimental results

Research questions

• RQ1 Can expert-guided multimodal fusion improve emotion recognition and sentiment analysis in a unified framework?
• RQ2 How do cross-modal interactions and multi-scale experts contribute to cross-lingual robustness between English and Chinese?
• RQ3 Is LoRA-based fine-tuning sufficient for multilingual multimodal affective tasks while maintaining efficiency?
• RQ4 Can a single generative framework handle both ERC classification and MSA regression effectively?

Key findings

• EGMF achieves state-of-the-art or competitive results across four bilingual datasets (MELD, CHERMA, MOSEI, SIMS-V2).
• On MOSEI, EGMF with GLM3-6B attains Acc-2 87.30, F1 87.09, Acc-7 55.38, MAE 0.496, Corr 0.801; SIMS-V2 shows Acc-2 81.56, F1 81.13, Acc2 (weak) 73.09, MAE 0.284, Corr 0.733.
• On MELD and CHERMA: EGMF-GLM3-6B achieves 65.57 WF1 on MELD and 73.90 WF1 on CHERMA (67.22 Acc and 65.57 WF1 for MELD; 73.97 Acc and 73.90 WF1 for CHERMA).
• Cross-lingual analysis shows stronger relative gains on Chinese datasets (+3.36 WF1 on CHERMA; +2.24 F1 on SIMS-V2) than English datasets.
• LoRA fine-tuning provides consistent English gains (0.74%-1.40%), with some degradation observed on Chinese datasets, highlighting language-specific adaptation needs.
• Ablation shows text modality is most critical; removing text drastically reduces performance; E1 and E3 (local and global experts) contribute prominently, with E2 providing steady gains.