[Paper Review] Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networks

Chin-Cheng Hsu, Hsin-Te Hwang+3 more|arXiv (Cornell University)|Apr 4, 2017

Speech Recognition and SynthesisComputer Science17 references153 citations

TL;DR

The paper proposes a non-parallel voice conversion framework that uses a conditional variational autoencoder (C-VAE) combined with a Wasserstein GAN (W-GAN) to synthesize target speech without frame alignment. It directly optimizes a VC loss using a VAW-GAN objective and yields more realistic spectra than a baseline VAE.

ABSTRACT

Building a voice conversion (VC) system from non-parallel speech corpora is challenging but highly valuable in real application scenarios. In most situations, the source and the target speakers do not repeat the same texts or they may even speak different languages. In this case, one possible, although indirect, solution is to build a generative model for speech. Generative models focus on explaining the observations with latent variables instead of learning a pairwise transformation function, thereby bypassing the requirement of speech frame alignment. In this paper, we propose a non-parallel VC framework with a variational autoencoding Wasserstein generative adversarial network (VAW-GAN) that explicitly considers a VC objective when building the speech model. Experimental results corroborate the capability of our framework for building a VC system from unaligned data, and demonstrate improved conversion quality.

Motivation & Objective

• Bridge the gap between parallel and non-parallel voice conversion by learning a unified generative speech model that does not require frame alignment.
• Infer phonetic content with a speaker-independent encoder while conditioning synthesis on speaker representations.
• Directly optimize voice conversion quality via a Wasserstein GAN objective integrated into the VAE framework.

Proposed method

• Formulate VC as a conditional VAE with a speaker-dependent decoder that combines a latent content z with a speaker representation y to reconstruct or convert speech.
• Enhance the C-VAE with a GAN objective by using a discriminator to distinguish real versus generated spectra (VAE-GAN idea).
• Adopt a Wasserstein distance objective to explicitly minimize the distance between true target distributions and converted distributions, via a 1-Lipschitz discriminator.
• Jointly train encoder, synthesizer, and discriminator in an alternating optimization scheme, with an objective that includes KL divergence, reconstruction loss, and W-GAN loss (J_vawgan).
• Use frame-by-frame spectral features (STRAIGHT SP, AP, F0) without frame alignment or post-filtering, and learn speaker representations as one-hot vectors embedded during generation.

Experimental results

Research questions

• RQ1 Can non-parallel voice conversion be effectively performed by a unified deep generative model without frame alignment?
• RQ2 Does incorporating a Wasserstein GAN objective into a VAE framework improve naturalness and spectral realism of converted speech compared to a baseline VAE?
• RQ3 How does directly optimizing VC loss influence the variance and spectral structure of converted speech?
• RQ4 What is the impact of using a speaker-conditioned synthesizer on cross-speaker conversion quality?

Key findings

• VAW-GAN outperforms the VAE baseline in human subjective naturalness (MOS) for inter- and intra-gender conversions.
• Converted spectral frames from VAW-GAN exhibit richer spectral envelopes with more pronounced frequency structure, contributing to clearer voices.
• The VAW-GAN outputs show higher spectral variance, suggesting the model does not collapse predictions to the mean as strongly as the VAE does.
• Frame-by-frame non-parallel VC is feasible with a conditional generator and a W-GAN objective, improving perceived quality over a vanilla VAE approach.
• The speaker similarity metric did not show clear improvement, indicating potential limitations of the single global speaker representation in capturing speaker traits.