[Paper Review] The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress

Lukas Christ, Shahin Amiriparian+10 more|arXiv (Cornell University)|Jun 23, 2022

Humor Studies and Applications4 citations

TL;DR

This paper presents the MuSe 2022 Multimodal Sentiment Analysis Challenge, introducing three novel datasets for humor detection, emotional reaction intensity prediction, and continuous stress emotion regression. Using multimodal features and LSTM-based models, it establishes strong baseline results, achieving 0.8480 AUC for humor detection, 0.2801 mean Pearson correlation for emotional reactions, and 0.4931 CCC for arousal in stress prediction.

ABSTRACT

The Multimodal Sentiment Analysis Challenge (MuSe) 2022 is dedicated to multimodal sentiment and emotion recognition. For this year's challenge, we feature three datasets: (i) the Passau Spontaneous Football Coach Humor (Passau-SFCH) dataset that contains audio-visual recordings of German football coaches, labelled for the presence of humour; (ii) the Hume-Reaction dataset in which reactions of individuals to emotional stimuli have been annotated with respect to seven emotional expression intensities, and (iii) the Ulm-Trier Social Stress Test (Ulm-TSST) dataset comprising of audio-visual data labelled with continuous emotion values (arousal and valence) of people in stressful dispositions. Using the introduced datasets, MuSe 2022 2022 addresses three contemporary affective computing problems: in the Humor Detection Sub-Challenge (MuSe-Humor), spontaneous humour has to be recognised; in the Emotional Reactions Sub-Challenge (MuSe-Reaction), seven fine-grained `in-the-wild' emotions have to be predicted; and in the Emotional Stress Sub-Challenge (MuSe-Stress), a continuous prediction of stressed emotion values is featured. The challenge is designed to attract different research communities, encouraging a fusion of their disciplines. Mainly, MuSe 2022 targets the communities of audio-visual emotion recognition, health informatics, and symbolic sentiment analysis. This baseline paper describes the datasets as well as the feature sets extracted from them. A recurrent neural network with LSTM cells is used to set competitive baseline results on the test partitions for each sub-challenge. We report an Area Under the Curve (AUC) of .8480 for MuSe-Humor; .2801 mean (from 7-classes) Pearson's Correlations Coefficient for MuSe-Reaction, as well as .4931 Concordance Correlation Coefficient (CCC) and .4761 for valence and arousal in MuSe-Stress, respectively.

Motivation & Objective

• To advance multimodal sentiment and emotion recognition by introducing three new, diverse datasets for humor, emotional reactions, and stress.
• To address contemporary challenges in affective computing: spontaneous humor detection, fine-grained emotion intensity prediction, and continuous stress-level regression.
• To provide a benchmark for multimodal fusion techniques across audio, video, text, and physiological signals in real-world, unscripted settings.
• To support research in audio-visual emotion recognition, health informatics, and symbolic sentiment analysis through open-access data and features.
• To establish transparent, reproducible baselines using deep learning and multimodal feature fusion for future method development.

Proposed method

• The challenge uses three distinct datasets: Passau-SFCH for humor detection, Hume-Reaction for emotional reaction intensity prediction, and Ulm-TSST for continuous stress-level regression.
• Features are extracted from audio (eGeMAPS, DeepSpectrum), video (VGGface2, facial action units), text (BERT), and physiological signals (BPM, ECG, RESP) using open-source tools.
• A recurrent neural network with LSTM cells is employed as the baseline model for all three sub-challenges to ensure consistency and comparability.
• Late fusion strategies combine audio, video, and text modalities, with separate models trained on individual and combined feature sets to optimize performance.
• For MuSe-Stress, physiological signals (EDA, ECG, RESP, BPM) are used as input, with labels derived from the MuSe-Physio 2021 challenge for consistency.
• Baseline models are trained on training splits and evaluated on development and test sets, with performance reported using standard metrics: AUC, Pearson correlation (ρ), and Concordance Correlation Coefficient (CCC).

Experimental results

Research questions

• RQ1 Can multimodal features effectively detect spontaneous humor in unscripted football press conferences?
• RQ2 To what extent can fine-grained emotional reaction intensities (e.g., amusement, fear, surprise) be predicted from self-recorded audio-visual responses to emotional stimuli?
• RQ3 How accurately can continuous valence and arousal levels be predicted in a stress-inducing social scenario using multimodal and physiological signals?
• RQ4 Which combination of modalities (audio, video, text, physiology) yields the strongest performance in multimodal sentiment and emotion recognition tasks?
• RQ5 How robust are baseline models across diverse, real-world recording conditions with variable noise and quality?

Key findings

• The MuSe-Humor sub-challenge achieved an Area Under the Curve (AUC) of 0.8480 using VGGface2 features, indicating strong performance in detecting humor in spontaneous coach press conferences.
• For the MuSe-Reaction sub-challenge, the mean Pearson correlation coefficient (ρ) across seven emotion classes was 0.2801, with FAU features showing competitive performance.
• In the MuSe-Stress sub-challenge, the model achieved a Concordance Correlation Coefficient (CCC) of 0.4931 for arousal and 0.4761 for valence using late fusion of audio and text features.
• Late fusion of audio and video modalities yielded the highest performance in MuSe-Stress (CCC = 0.6914 for valence), outperforming individual modality baselines.
• The use of physiological signals (BPM, ECG, RESP) improved baseline results in MuSe-Stress, with a CCC of 0.4361 for valence using combined physiological features.
• The inclusion of raw data, extracted features, and code in public repositories enables reproducibility and facilitates future research in multimodal affective computing.