[Paper Review] The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systems

Adaeze Adigwe, Noé Tits+3 more|arXiv (Cornell University)|Jun 25, 2018

Speech Recognition and SynthesisComputer Science18 references51 citations

TL;DR

This paper introduces an open-source multi-speaker, bilingual emotional speech database (English and French) aimed at enabling controllable emotion generation in voice synthesis and transformation; it validates the dataset with a simple emotion-to-neutral transformation using a neural network and CMOS testing.

ABSTRACT

In this paper, we present a database of emotional speech intended to be open-sourced and used for synthesis and generation purpose. It contains data for male and female actors in English and a male actor in French. The database covers 5 emotion classes so it could be suitable to build synthesis and voice transformation systems with the potential to control the emotional dimension in a continuous way. We show the data's efficiency by building a simple MLP system converting neutral to angry speech style and evaluate it via a CMOS perception test. Even though the system is a very simple one, the test show the efficiency of the data which is promising for future work.

Motivation & Objective

• Provide an open-source emotional speech dataset suitable for deep learning-based synthesis and voice transformation.
• Cover multiple speakers (both genders) and languages (English and French) to support cross-language emotion modeling.
• Support continuous emotion control by placing data within a dimensional (Russell circumplex) or basic-emotion framework.
• Validate dataset utility through a baseline voice transformation experiment and perceptual evaluation.

Proposed method

• Assemble a multilingual, multi-speaker emotional speech database by reusing phonetically balanced sentences from CMU Arctic (English) and SIWIS (French).
• Recordings in 44.1 kHz, downsampled to 16 kHz, 16-bit PCM in anechoic rooms; include five emotions (amused, anger, sleepiness, disgust, neutral).
• Use manual segmentation and annotations; provide transcriptions and phonetic-level annotations (not time-aligned yet).
• Build a simple feedforward neural network (6 hidden layers of 1024 tanh units) within the Merlin toolkit to perform neutral-to-emotional voice conversion for target emotion (anger).
• Train per-speaker, per-emotion voice transformation models using WORLD vocoder features and DTW alignment.
• Evaluate with a Comparative Mean Opinion Score (CMOS) test and emotion-categorization judgments to assess perceptual realism and emotion recognizability.

Experimental results

Research questions

• RQ1 Can an open-source, multi-speaker emotional speech database support training of neural models for emotion-controlled voice generation?
• RQ2 How recognizable and controllable is the anger emotion when transforming neutral speech in a baseline system?
• RQ3 Does aligning and transforming within a constrained emotion set (including neutral) yield perceptually valid emotional speech?
• RQ4 To what extent can the dataset integrate with existing open resources to enable cross-language and cross-speaker emotion modeling?

Key findings

• The database provides data for five emotions across English (two female, two male) and French (one male) and is suitable for deep learning-based synthesis and voice transformation.
• A simple DNN-based voice transformation trained per speaker demonstrates effective neutral-to-anger transformation, with high emotion recognition in perception tests.
• CMOS results show that angry speech is generally correctly identified as more emotional than neutral, with varying accuracy across speakers.
• Perception tests indicate most misclassifications involve neutral speech being perceived as sleepiness or amusement rather than anger.
• The data and methodology suggest promising potential for future, more complex synthesis systems (e.g., attention-based or WaveNet-like architectures).