[Paper Review] Emotion Recognition From Speech With Recurrent Neural Networks

V. K. Chernykh, Prikhodko, Pavel|arXiv (Cornell University)|Jan 27, 2017

Speech and Audio Processing32 references127 citations

TL;DR

The paper presents a recurrent neural network approach with Connectionist Temporal Classification (CTC) loss for emotion recognition from speech, evaluated on IEMOCAP and compared to baselines and human performance.

ABSTRACT

In this paper the task of emotion recognition from speech is considered. Proposed approach uses deep recurrent neural network trained on a sequence of acoustic features calculated over small speech intervals. At the same time special probabilistic-nature CTC loss function allows to consider long utterances containing both emotional and neutral parts. The effectiveness of such an approach is shown in two ways. Firstly, the comparison with recent advances in this field is carried out. Secondly, human performance on the same task is measured. Both criteria show the high quality of the proposed method.

Motivation & Objective

• Address the challenge of recognizing emotion in long utterances where emotions may be localised within speech.
• Leverage a deep recurrent neural network with CTC loss to align frame-level features to utterance-level emotion labels.
• Compare CTC-based sequence-to-sequence learning with framewise and one-label baselines under speaker-independent cross-validation.
• Assess the quality of the proposed method relative to human performance on the same task.

Proposed method

• Preprocess audio into 200 ms frames with 100 ms overlap and extract 34 acoustic features per frame (MFCCs, chroma, energy, etc.).
• Use a Bidirectional LSTM based neural network to output probabilities over an extended label set including NULL.
• Train with Connectionist Temporal Classification (CTC) loss to handle unaligned long input sequences and produce a sequence of emotion labels.
• Compare CTC to framewise and one-label baselines under grouped cross-validation to ensure speaker-independence.
• Evaluate using overall accuracy and mean class accuracy to address class imbalance.

Experimental results

Research questions

• RQ1 Can a CTC-based RNN model accurately recognise utterance-level emotions from frame-level acoustic features in long speech sequences?
• RQ2 How does the CTC approach compare to framewise and one-label baselines on the IEMOCAP dataset under speaker-independent evaluation?
• RQ3 What is the impact of subjectivity and label consistency among human raters on model performance?
• RQ4 How close is machine performance to human performance on the emotion recognition task?

Key findings

• CTC achieves 54% overall accuracy and 54% mean class accuracy, outperforming framewise (45%/41%) and one-label (51%/49%) baselines, and approaching human performance (69%/70%).
• Framewise baseline shows instability for longer utterances, while CTC maintains robust performance.
• Human raters achieve higher accuracy than the model, highlighting subjective aspects of emotion labeling and room for improvement.
• The model’s error patterns partly align with human labeling inconsistencies, suggesting the model captures perceptual cues similar to humans.
• The approach demonstrates the viability of end-to-end sequence modelling for emotion recognition using CTC in speech.