[Paper Review] Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understanding

Xiaojiang Peng, Yutao Chen+12 more|arXiv (Cornell University)|Jan 23, 2026

Emotion and Mood Recognition0 citations

TL;DR

Emotion-LLaMAv2 proposes an end-to-end multimodal emotion understanding framework with a Conv Attention pre-fusion module and a perception-to-cognition curriculum, evaluated on the unified MMEVerse benchmark. It achieves state-of-the-art results and better generalization over open-source MLLMs.

ABSTRACT

Understanding human emotions from multimodal signals poses a significant challenge in affective computing and human-robot interaction. While multimodal large language models (MLLMs) have excelled in general vision-language tasks, their capabilities in emotional reasoning remain limited. The field currently suffers from a scarcity of large-scale datasets with high-quality, descriptive emotion annotations and lacks standardized benchmarks for evaluation. Our preliminary framework, Emotion-LLaMA, pioneered instruction-tuned multimodal learning for emotion reasoning but was restricted by explicit face detectors, implicit fusion strategies, and low-quality training data with limited scale. To address these limitations, we present Emotion-LLaMAv2 and the MMEVerse benchmark, establishing an end-to-end pipeline together with a standardized evaluation setting for emotion recognition and reasoning. Emotion-LLaMAv2 introduces three key advances. First, an end-to-end multiview encoder eliminates external face detection and captures nuanced emotional cues via richer spatial and temporal multiview tokens. Second, a Conv Attention pre-fusion module is designed to enable simultaneous local and global multimodal feature interactions external to the LLM backbone. Third, a perception-to-cognition curriculum instruction tuning scheme within the LLaMA2 backbone unifies emotion recognition and free-form emotion reasoning. To support large-scale training and reproducible evaluation, MMEVerse aggregates twelve publicly available emotion datasets, including IEMOCAP, MELD, DFEW, and MAFW, into a unified multimodal instruction format. The data are re-annotated via a multi-agent pipeline involving Qwen2 Audio, Qwen2.5 VL, and GPT 4o, producing 130k training clips and 36k testing clips across 18 evaluation benchmarks.

Motivation & Objective

• Motivate robust multimodal emotion understanding that couples perception with semantic reasoning across audio, visual, and textual signals.
• Eliminate dependence on external face detectors to enable end-to-end training and richer emotion cues.
• Unify emotion recognition and emotion reasoning within a language-model framework through curriculum instruction tuning.
• Provide a large-scale, standardized benchmark (MMEVerse) for reproducible evaluation across diverse datasets and tasks.

Proposed method

• Develop an end-to-end multimodal encoder with a multiview visual and an audio encoder to capture spatial, temporal, and prosodic cues.
• Introduce a Conv Attention pre-fusion module that enables simultaneous local and global cross-modal interactions before LLM input.
• Align fused multimodal representations to the LLM space via a modal adapter for LoRA-tuned instruction-following with emotion tasks.
• Employ a perception-to-cognition curriculum that stages learning from basic emotion recognition to context-aware emotion reasoning within the LLaMA2 backbone.
• Construct MMEVerse by aggregating 12 datasets into a unified instruction-tuning format and re-annotating with a multi-agent pipeline to produce 130k training clips and 36k testing clips.

Experimental results

Research questions

• RQ1 How can end-to-end multimodal emotion understanding be achieved without explicit face detectors?
• RQ2 Can a Conv Attention pre-fusion module improve cross-modal interactions for emotion perception?
• RQ3 Does curriculum-based instruction tuning improve both emotion recognition and reasoning in a unified LLM framework?
• RQ4 Is a large-scale, standardized benchmark like MMEVerse effective for training and evaluating multimodal emotion models across diverse datasets?

Key findings

• Emotion-LLaMAv2 outperforms representative open-source MLLMs on MER-UniBench and MMEVerse-Bench.
• The model shows improved generalization and more structured multimodal reasoning behavior.
• MMEVerse provides a unified, scalable resource with 129k training clips and 36k testing clips across 18 benchmarks.
• Emotion-LLaMAv2 achieves competitive or superior results compared with Qwen2.5 Omni, HumanOmni, and AffectGPT.
• Ablation studies show benefits from end-to-end encoding, Conv Attention fusion, and perception-to-cognition curriculum.