[Paper Review] ERM-MinMaxGAP: Benchmarking and Mitigating Gender Bias in Multilingual Multimodal Speech-LLM Emotion Recognition

Zi Haur Pang, Xiaoxue Gao+2 more|arXiv (Cornell University)|Mar 22, 2026

Emotion and Mood Recognition0 citations

TL;DR

Introduces MELD-ST based multilingual multimodal SER benchmark and proposes ERM-MinMaxGAP, a fairness-aware training objective that reduces worst-language gender gaps while improving SER performance using adaptive fairness weighting and a MinMaxGAP regularizer.

ABSTRACT

Speech emotion recognition (SER) systems can exhibit gender-related performance disparities, but how such bias manifests in multilingual speech LLMs across languages and modalities is unclear. We introduce a novel multilingual, multimodal benchmark built on MELD-ST, spanning English, Japanese, and German, to quantify language-specific SER performance and gender gaps. We find bias is strongly language-dependent, and multimodal fusion does not reliably improve fairness. To address these, we propose ERM-MinMaxGAP, a fairness-informed training objective, which augments empirical risk minimization (ERM) with a proposed adaptive fairness weight mechanism and a novel MinMaxGAP regularizer on the maximum male-female loss gap within each language and modality. Building upon the Qwen2-Audio backbone, our ERM-MinMaxGAP approach improves multilingual SER performance by 5.5% and 5.0% while reducing the overall gender bias gap by 0.1% and 1.4% in the unimodal and multimodal settings, respectively.

Motivation & Objective

• Motivate and quantify gender bias in multilingual multimodal speech LLM-based SER across English, Japanese, and German.
• Provide a controlled benchmark (MELD-ST) to disentangle language effects from corpus effects on SER fairness.
• Propose ERM-MinMaxGAP, a training objective combining ERM with a Max-language gender-gap regularizer to reduce worst-case disparities.
• Demonstrate that multimodal fusion does not consistently improve fairness and show improved performance–fairness trade-offs with the proposed method.

Proposed method

• Start from supervised fine-tuning of a speech LLM backbone (Qwen2-Audio) with LoRA.
• Define within-language gender loss gaps and the MaxGAP regularizer: Delta_l(θ) = |L_{l,F}(θ) − L_{l,M}(θ)| and Δ_max(θ) = max_l Δ_l(θ).
• Introduce MinMaxGAP regularizer R_MinMaxGAP(θ) = (Δ_max(θ))^p with p ∈ {1,2} (empirically p=2).
• Incorporate an adaptive fairness weight via a Lagrange-multiplier inspired update to λ: λ^{(k+1)} = Π_[0,λ_max](λ^{(k)} + η(Δ_dev^{(k)} − ε)).
• Combine objectives: L_{ERM−MinMaxGAP}^{(k)}(θ) = L_{ERM}(θ) + λ^{(k)} R_MinMaxGAP(θ).
• Evaluate both unimodal and multimodal inputs on MELD-ST with English, Japanese, German, reporting SER (W-F1, ACC) and gender bias gaps (TPR, FPR, W-F1, ACC) and AVG.

Figure 1: Architecture of the proposed method. The method consists of (1) empirical risk minimization for overall SER improvement, (2) MinMaxGAP for minimizing the language-wise gender gap, and (3) adaptive fairness-weight adjustment for fairness-aware SER.

Experimental results

Research questions

• RQ1 How does gender bias manifest in multilingual, multimodal SER using speech LLM backbones across English, Japanese, and German?
• RQ2 Does multimodal fusion reduce gender bias consistently across languages and settings?
• RQ3 Can a fairness-aware training objective reduce worst-language gender gaps without sacrificing SER performance?
• RQ4 What is the effectiveness of an adaptive fairness weight in balancing task performance and fairness during training?

Key findings

• Gender bias in multilingual multimodal SER is highly language- and model-dependent; multimodal inputs do not reliably reduce gender disparities.
• ERM-MinMaxGAP improves overall SER performance versus baselines in both unimodal and multimodal settings (e.g., multilingual gains of +5.49 W-F1 and +9.75 ACC unimodal; +5.03 W-F1 and +3.62 ACC multimodal) while reducing the AVG gender bias.
• The MinMaxGAP regularizer strengthens the performance–fairness trade-off by reducing the worst-language gender gap across languages and modalities.
• Adaptive fairness weighting with a constrained optimization-inspired update yields better fairness–utility balance than fixed regularization strength.
• Ablation shows ERM + MinMaxGAP outperforms zero-shot baselines, and adaptive λ with p=2 provides favorable fairness and SER results compared to p=1 or fixed λ settings.