[Paper Review] SEGAA: A Unified Approach to Predicting Age, Gender, and Emotion in Speech

Raluca Anca Corb Aron, Indra Sigicharla+5 more|arXiv (Cornell University)|Mar 1, 2024

Emotion and Mood Recognition4 citations

TL;DR

This paper proposes SEGAA, a multi-output deep learning model that jointly predicts age, gender, and emotion from speech using a unified architecture. By leveraging a shared feature extractor and task-specific heads, SEGAA achieves competitive performance—reaching 99% accuracy in gender detection—while efficiently modeling interdependencies between variables and outperforming sequential and individual models in runtime efficiency and error stability.

ABSTRACT

The interpretation of human voices holds importance across various applications. This study ventures into predicting age, gender, and emotion from vocal cues, a field with vast applications. Voice analysis tech advancements span domains, from improving customer interactions to enhancing healthcare and retail experiences. Discerning emotions aids mental health, while age and gender detection are vital in various contexts. Exploring deep learning models for these predictions involves comparing single, multi-output, and sequential models highlighted in this paper. Sourcing suitable data posed challenges, resulting in the amalgamation of the CREMA-D and EMO-DB datasets. Prior work showed promise in individual predictions, but limited research considered all three variables simultaneously. This paper identifies flaws in an individual model approach and advocates for our novel multi-output learning architecture Speech-based Emotion Gender and Age Analysis (SEGAA) model. The experiments suggest that Multi-output models perform comparably to individual models, efficiently capturing the intricate relationships between variables and speech inputs, all while achieving improved runtime.

Motivation & Objective

• To address the gap in joint prediction of age, gender, and emotion from speech using a single, unified model.
• To overcome the limitations of sequential models, which propagate errors across stages, by introducing a multi-output architecture.
• To improve efficiency and predictive performance by modeling interdependencies between age, gender, and emotion in a shared representation.
• To demonstrate that multi-output learning can match or exceed individual model performance while reducing computational overhead.
• To create a robust, scalable framework for real-world applications in healthcare, customer service, and mental health monitoring.

Proposed method

• The SEGAA model employs a shared convolutional neural network (CNN) backbone to extract shared speech representations from raw audio inputs.
• Three separate, task-specific heads are attached to the shared feature extractor: one for age regression, one for gender classification, and one for emotion classification.
• The model is trained end-to-end using a multi-task loss function combining mean squared error (MSE) for age, categorical cross-entropy for gender, and cross-entropy for emotion.
• The Nadam optimizer is used to improve convergence and generalization, particularly in the multi-output setting.
• The training data is constructed by merging the CREMA-D and EMO-DB datasets, ensuring all three labels (age, gender, emotion) are present.
• Model performance is evaluated using accuracy, precision, recall, and F1-score across all three tasks, with confusion matrices used to analyze error patterns.

Figure 1: Workflow Diagram

Experimental results

Research questions

• RQ1 Can a unified multi-output deep learning model achieve comparable or better performance than individual models in predicting age, gender, and emotion from speech?
• RQ2 How does error propagation in sequential models (e.g., emotion → gender → age) affect overall prediction accuracy compared to joint prediction?
• RQ3 To what extent do interdependencies between age, gender, and emotion enhance model performance when modeled jointly?
• RQ4 Does the proposed multi-output architecture maintain high efficiency and low latency compared to individual or sequential models?
• RQ5 How does the model perform on imbalanced age groups, particularly in underrepresented demographics like those in their 60s and 70s?

Key findings

• The multi-output SEGAA model achieved 99% accuracy in gender detection, outperforming individual models and demonstrating strong generalization.
• For emotion prediction, the multi-output SEGAA model achieved 96% accuracy, with slightly lower performance on 'neutral' and 'anxiety/fear' classes due to class imbalance.
• The model maintained high performance on age prediction, though accuracy was slightly reduced for older age groups due to data scarcity in the 60s and 70s.
• Compared to sequential models (e.g., Emotion → Gender → Age), the multi-output SEGAA model showed significantly lower error propagation and better overall F1 scores.
• Despite marginal superiority in accuracy of individual models, the multi-output SEGAA model offered superior runtime efficiency and stability, making it more suitable for real-time applications.
• Confusion matrices confirmed that both individual and multi-output SEGAA models exhibited low misclassification rates, with the multi-output version showing consistent performance across all classes.

Figure 2: Model architecture for Individual MLP