[Paper Review] StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networks

Hirokazu Kameoka, Takuhiro Kaneko+2 more|arXiv (Cornell University)|Jun 6, 2018

Speech Recognition and Synthesis37 references48 citations

TL;DR

StarGAN-VC enables non-parallel many-to-many voice conversion using a single generator conditioned on target attributes, achieving real-time performance with minutes of training data and outperforming VAE-GAN baselines in subjective tests.

ABSTRACT

This paper proposes a method that allows non-parallel many-to-many voice conversion (VC) by using a variant of a generative adversarial network (GAN) called StarGAN. Our method, which we call StarGAN-VC, is noteworthy in that it (1) requires no parallel utterances, transcriptions, or time alignment procedures for speech generator training, (2) simultaneously learns many-to-many mappings across different attribute domains using a single generator network, (3) is able to generate converted speech signals quickly enough to allow real-time implementations and (4) requires only several minutes of training examples to generate reasonably realistic-sounding speech. Subjective evaluation experiments on a non-parallel many-to-many speaker identity conversion task revealed that the proposed method obtained higher sound quality and speaker similarity than a state-of-the-art method based on variational autoencoding GANs.

Motivation & Objective

• Enable non-parallel voice conversion across multiple target attributes (many-to-many) with no parallel utterances or alignments.
• Use a single generator network to model mappings between all attribute domains efficiently.
• Achieve real-time or near-real-time speech conversion suitable for practical deployment.
• Demonstrate improved subjective speech quality and speaker similarity over a VAE-GAN baseline.

Proposed method

• Adopt StarGAN to learn multiple domain mappings with a single encoder–decoder generator G conditioned on a target attribute c.
• Train a real/fake discriminator D and a domain classifier C to ensure converted speech is realistic and belongs to the target attribute.
• Use adversarial loss L_adv, domain classification loss L_cls, cycle consistency loss L_cyc, and identity loss L_id to regularize mappings (Equations 9–17).
• Represent attributes as concatenated one-hot vectors and model G as a fully convolutional CNN with a gated linear unit (GLU) in an encoder–decoder structure.
• Employ a PatchGAN-like discriminator D and a domain classifier C that operate on acoustic feature sequences (mel-cepstral coefficients) to produce segment-wise real/fake and class probabilities.
• Convert sequences using mel-cepstral features via WORLD, with spectral gain-based time-domain reconstruction through a vocoder (Section 3.2).

Experimental results

Research questions

• RQ1 Can non-parallel, many-to-many voice conversion be achieved with a single generator using StarGAN?
• RQ2 Does StarGAN-VC preserve linguistic content while enabling diverse attribute transfers across multiple speakers?
• RQ3 Is the method capable of real-time conversion and does it require only a few minutes of training data?
• RQ4 How does StarGAN-VC compare to non-parallel VC approaches such as CVAE-VC or CycleGAN-VC in terms of quality and similarity?

Key findings

• Subjective evaluations show StarGAN-VC yields higher sound quality than a VAE-GAN baseline.
• StarGAN-VC achieves better speaker similarity to target speakers in ABX tests compared with the baseline.
• The approach supports non-parallel many-to-many conversions using a single generator across multiple speaker attributes.
• The method operates with real-time-like efficiency and requires only several minutes of training data.
• Experiments on the VCC 2018 dataset with four target speakers demonstrate significant performance gains in perceived quality and similarity.