[Paper Review] Large Raw Emotional Dataset with Aggregation Mechanism

В. В. Кондратенко, Artem Sokolov+4 more|arXiv (Cornell University)|Dec 23, 2022

Speech Recognition and Synthesis4 citations

TL;DR

This paper introduces Dusha, a large-scale, open-source, bi-modal Russian speech emotion recognition dataset comprising ~350 hours of audio and transcripts from acted (crowd-sourced) and real-life (podcast) sources. It proposes a Dawid-Skene-based aggregation mechanism with a 0.9 confidence threshold to generate reliable emotion labels, enabling both pre-training (acted) and fine-tuning (realistic) applications, with a baseline model achieving 0.83 UA and 0.77 F1 on the acted test set.

ABSTRACT

We present a new data set for speech emotion recognition (SER) tasks called Dusha. The corpus contains approximately 350 hours of data, more than 300 000 audio recordings with Russian speech and their transcripts. Therefore it is the biggest open bi-modal data collection for SER task nowadays. It is annotated using a crowd-sourcing platform and includes two subsets: acted and real-life. Acted subset has a more balanced class distribution than the unbalanced real-life part consisting of audio podcasts. So the first one is suitable for model pre-training, and the second is elaborated for fine-tuning purposes, model approbation, and validation. This paper describes pre-processing routine, annotation, and experiment with a baseline model to demonstrate some actual metrics which could be obtained with the Dusha data set.

Motivation & Objective

• To address the scarcity of large-scale, diverse, and realistic speech emotion datasets in Russian by creating a new open benchmark.
• To mitigate bias and improve label reliability in emotion recognition datasets through a multi-annotator crowd-sourcing approach with statistical aggregation.
• To provide a dual-domain dataset—acted (balanced) and real-life (unbalanced)—to support both pre-training and real-world model evaluation.
• To release a complete pipeline including data, preprocessing scripts, and a baseline model to lower the entry barrier for SER research.
• To enable cross-lingual and transfer learning studies by offering a high-quality, multilingual-ready corpus in a low-resource language (Russian).

Proposed method

• The dataset, named Dusha, collects approximately 350 hours of Russian speech from two sources: crowd-sourced acted recordings (255 hours) and real podcast recordings (90 hours).
• Each audio sample is paired with a transcript, forming a bi-modal dataset suitable for multimodal emotion recognition.
• Emotions are labeled using four classes: Anger, Happiness, Neutral, and Sadness, with each sample annotated by multiple crowdworkers.
• A Dawid-Skene (DS) algorithm with a confidence threshold of 0.9 is applied to aggregate individual annotations into a single, high-confidence label per sample.
• The aggregation process supports both single-label and multi-label outputs, enhancing label reliability and reducing annotation bias.
• A baseline model is implemented using Mel-spectrograms (64 filterbanks, 20ms window, 10ms overlap), a MobileNetV2-based architecture with a self-attention layer, and trained via Adam optimizer for 100 epochs with a batch size of 64.

Experimental results

Research questions

• RQ1 Can a large-scale, bi-modal speech emotion dataset in Russian be constructed with sufficient diversity and realism to support robust SER model training and evaluation?
• RQ2 How effective is the Dawid-Skene aggregation mechanism with a 0.9 confidence threshold in improving label reliability compared to raw crowd-sourced annotations?
• RQ3 To what extent can a baseline model trained on the acted subset of Dusha generalize to real-world, unstructured audio from podcasts?
• RQ4 What performance metrics (e.g., F1, accuracy) can be achieved on the Dusha benchmark using standard deep learning architectures and standard evaluation protocols?
• RQ5 How does the performance of a model trained on Dusha compare to state-of-the-art models trained on established benchmarks like IEMOCAP4?

Key findings

• The Dusha dataset contains 230,177 training samples and 24,672 test samples after aggregation, spanning approximately 260 hours of training and 28 hours of testing audio.
• The acted (crowd-sourced) subset contains 164,387 samples with a relatively balanced distribution across the four emotion classes, while the podcast subset (90,462 samples) shows a strong dominance of neutral emotion.
• The baseline model achieves 0.83 unweighted accuracy (UA) and 0.77 macro F1 score on the acted test set, indicating strong performance on the balanced subset.
• On the more realistic podcast test set, the model achieves 0.89 UA but only 0.53 weighted accuracy (WA) and 0.54 F1, reflecting the challenge of domain shift and class imbalance.
• The model trained on IEMOCAP4 achieves 0.59 UA, 0.59 WA, and 0.59 F1 on 5-fold cross-validation, serving as a baseline for comparison with Dusha results.
• The aggregation mechanism successfully reduces noise in crowd-sourced labels, with 90% of samples achieving a confidence score above 0.9, ensuring high-quality label outputs for downstream tasks.