[Paper Review] An Ensemble Framework of Voice-Based Emotion Recognition System for Films and TV Programs

Fei Tao, Gang Liu+1 more|arXiv (Cornell University)|Mar 3, 2018

Music and Audio Processing21 references3 citations

TL;DR

This paper proposes an ensemble deep learning framework for voice-based emotion recognition in real-world film and TV audio, combining handcrafted features, iVector, lexical information, and attention-based RNNs with multi-task learning for gender and speaker classification. The system achieves a 29.5% relative improvement (7.8 percentage points absolute) over the best baseline, demonstrating robustness in noisy, non-speech-rich, and variable-length audio scenarios.

ABSTRACT

Employing voice-based emotion recognition function in artificial intelligence (AI) product will improve the user experience. Most of researches that have been done only focus on the speech collected under controlled conditions. The scenarios evaluated in these research were well controlled. The conventional approach may fail when background noise or nonspeech filler exist. In this paper, we propose an ensemble framework combining several aspects of features from audio. The framework incorporates gender and speaker information relying on multi-task learning. Therefore it is able to dig and capture emotional information as much as possible. This framework is evaluated on multimodal emotion challenge (MEC) 2017 corpus which is close to real world. The proposed framework outperformed the best baseline system by 29.5% (relative improvement).

Motivation & Objective

• To develop a robust voice-based emotion recognition system that performs well in real-world audio conditions, such as those with background noise, non-speech fillers, and variable segment lengths.
• To address the limitations of prior work that relies on controlled studio data by evaluating on the MEC 2017 corpus, which consists of clips from films and TV programs with realistic audio challenges.
• To improve emotion recognition performance by integrating multiple feature types—handcrafted prosodic features (IS10), iVector representations, lexical content from ASR, and deep neural representations via attention-based RNNs.
• To leverage multi-task learning with gender and speaker classification as auxiliary tasks to enhance the main emotion recognition task, using labels that are easily obtainable in practice.
• To design an ensemble framework that fuses multiple sub-systems to exploit complementary strengths and improve overall recognition accuracy and robustness.

Proposed method

• The framework employs an ensemble of four sub-systems: multi-task DNN using IS10 features, multi-task DNN using iVector features, lexical SVM based on ASR transcripts, and an attention-based weighted pooling RNN.
• Multi-task learning is applied in the DNN and RNN sub-systems, where emotion classification is the main task and gender and speaker classification are auxiliary tasks with task weights of 0.6 and 0.3, respectively.
• The attention mechanism in the RNN computes context-aware pooling using a learned weight vector: $ A_T = \frac{\exp(W \cdot h_T)}{\sum_{t=1}^{n} \exp(W \cdot h_T)} $, allowing the model to focus on salient temporal segments.
• Feature normalization is applied: IS10 features are z-normalized using training set statistics, and sequential features are normalized within each utterance.
• The ensemble fusion combines sub-system predictions using a linear combination weight trained on validation data to optimize macro F-score.
• The system is trained using SGD for DNNs and Adam for RNNs, with a batch size of 32 and 10% of training data reserved for validation.

Experimental results

Research questions

• RQ1 Can an ensemble framework combining diverse audio features (prosodic, iVector, lexical, deep representations) outperform single-model baselines in real-world emotion recognition?
• RQ2 How effective is multi-task learning with gender and speaker classification as auxiliary tasks for improving emotion recognition in low-resource, noisy, and non-speech-rich audio?
• RQ3 To what extent do non-speech vocalizations (e.g., sighs, laughs) contribute to emotion recognition, and can they be effectively modeled in the proposed framework?
• RQ4 Does attention-based RNN with weighted pooling improve performance on variable-length utterances compared to standard RNNs or DNNs?
• RQ5 How does the fusion of multiple sub-systems with complementary strengths enhance overall emotion recognition accuracy and robustness on the MEC 2017 corpus?

Key findings

• The proposed ensemble framework achieved a macro F-score (MAF) of 34.2%, representing a 7.8 percentage point absolute improvement over the best baseline system (DNN with single task), or a 29.5% relative improvement.
• The multi-task DNN using IS10 features achieved a MAF of 32.4%, outperforming the single-task DNN baseline (26.4%) by 6 percentage points, demonstrating the benefit of auxiliary tasks.
• The attention-based RNN sub-system achieved a MAF of 23.2%, which was lower than the baseline DNN, likely due to insufficient training data (4.5 hours) for effective RNN training.
• The lexical SVM sub-system performed poorly (MAF 17.7%), indicating that ASR-derived text features were unreliable in this context, possibly due to transcription errors.
• The multi-task DNN using iVector features achieved a MAF of 27.4%, showing that iVector representations are also effective for emotion recognition, though less so than IS10 features.
• The ensemble fusion strategy successfully combined complementary strengths of sub-systems, confirming that diverse feature types and modeling approaches enhance overall performance on real-world audio.