[Paper Review] MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition

Zheng Lian, Licai Sun+6 more|arXiv (Cornell University)|Jan 7, 2024

Emotion and Mood Recognition6 citations

TL;DR

MERBench provides a unified evaluation benchmark for multimodal emotion recognition and introduces MER2023, a Chinese emotion dataset, to enable fair comparisons and robustness analyses across methods.

ABSTRACT

Multimodal emotion recognition plays a crucial role in enhancing user experience in human-computer interaction. Over the past few decades, researchers have proposed a series of algorithms and achieved impressive progress. Although each method shows its superior performance, different methods lack a fair comparison due to inconsistencies in feature extractors, evaluation manners, and experimental settings. These inconsistencies severely hinder the development of this field. Therefore, we build MERBench, a unified evaluation benchmark for multimodal emotion recognition. We aim to reveal the contribution of some important techniques employed in previous works, such as feature selection, multimodal fusion, robustness analysis, fine-tuning, pre-training, etc. We hope this benchmark can provide clear and comprehensive guidance for follow-up researchers. Based on the evaluation results of MERBench, we further point out some promising research directions. Additionally, we introduce a new emotion dataset MER2023, focusing on the Chinese language environment. This dataset can serve as a benchmark dataset for research on multi-label learning, noise robustness, and semi-supervised learning. We encourage the follow-up researchers to evaluate their algorithms under the same experimental setup as MERBench for fair comparisons. Our code is available at: https://github.com/zeroQiaoba/MERTools.

Motivation & Objective

• Provide a unified, fair evaluation framework for multimodal emotion recognition across datasets, features, and fusion strategies.
• Reproduce and compare diverse methods under a consistent experimental setup to reveal the impact of features, fusion, pre-training, and fine-tuning.
• Introduce MER2023 as a Chinese emotion dataset to support multi-label learning, noise robustness, and semi-supervised learning benchmarks.
• Guide future research directions based on evaluation results and promote open, reproducible experiments.

Proposed method

• Reproduce representative unimodal and multimodal emotion recognition methods under a unified MERBench setup.
• Evaluate features across acoustic, lexical, and visual modalities to assess their contribution to performance.
• Implement multimodal fusion with modality importance weights and end-to-end training for discrete emotion and valence prediction.
• Use a joint loss combining cross-entropy for emotion classification and MSE for valence regression.
• Introduce MER2023 with four subsets (Train&Val, MER-MULTI, MER-NOISE, MER-SEMI) and design sampling, annotation, and splitting procedures.
• Provide an open-source toolchain (MERTools) to enable reproducible benchmarking.

Experimental results

Research questions

• RQ1 How to select appropriate features for different multimodal emotion datasets under a unified setup?
• RQ2 Which multimodal fusion strategies best leverage complementary cues across modalities?
• RQ3 How does cross-corpus performance behave under standardized evaluation, and how can robustness be improved?
• RQ4 What is the impact of pre-training and fine-tuning of feature extractors on downstream emotion recognition performance?
• RQ5 How can multi-label, noise-robustness, and semi-supervised learning be effectively studied in a Chinese-language emotion dataset?

Key findings

• MERBench enables fair, comprehensive comparisons by reproducing methods under a shared experimental framework.
• MER2023 provides structured subsets for exploring multi-label relationships, noise robustness, and semi-supervised learning in Chinese emotion recognition.
• Multimodal fusion with learned modality importance improves performance across datasets and settings compared to unimodal baselines.
• The benchmark highlights the influence of feature choice and pre-training/fine-tuning on cross-corpus and robustness performance.
• The authors provide open-source code to facilitate reproducible evaluation under MERBench conditions.