[Paper Review] Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networks

Takuhiro Kaneko, Hirokazu Kameoka|arXiv (Cornell University)|Nov 30, 2017

Speech Recognition and SynthesisComputer Science32 references179 citations

TL;DR

The paper introduces CycleGAN-VC, a parallel-data-free voice conversion method using cycle-consistent GANs with gated CNNs and an identity-mapping loss to map source to target speech without parallel data, reducing over-smoothing.

ABSTRACT

We propose a parallel-data-free voice-conversion (VC) method that can learn a mapping from source to target speech without relying on parallel data. The proposed method is general purpose, high quality, and parallel-data free and works without any extra data, modules, or alignment procedure. It also avoids over-smoothing, which occurs in many conventional statistical model-based VC methods. Our method, called CycleGAN-VC, uses a cycle-consistent adversarial network (CycleGAN) with gated convolutional neural networks (CNNs) and an identity-mapping loss. A CycleGAN learns forward and inverse mappings simultaneously using adversarial and cycle-consistency losses. This makes it possible to find an optimal pseudo pair from unpaired data. Furthermore, the adversarial loss contributes to reducing over-smoothing of the converted feature sequence. We configure a CycleGAN with gated CNNs and train it with an identity-mapping loss. This allows the mapping function to capture sequential and hierarchical structures while preserving linguistic information. We evaluated our method on a parallel-data-free VC task. An objective evaluation showed that the converted feature sequence was near natural in terms of global variance and modulation spectra. A subjective evaluation showed that the quality of the converted speech was comparable to that obtained with a Gaussian mixture model-based method under advantageous conditions with parallel and twice the amount of data.

Motivation & Objective

• Motivate and address the need for voice conversion without parallel data or extra alignment modules.
• Develop a general, high-quality VC approach that avoids over-smoothing inherent in traditional methods.
• Leverage CycleGANs to learn forward and inverse mappings from unpaired data while preserving linguistic information.
• Demonstrate that CycleGAN-VC can achieve near-natural feature conversion without parallel data on VCC 2016.

Proposed method

• Use CycleGAN with forward and inverse mappings (G_X->Y and G_Y->X) trained with adversarial and cycle-consistency losses.
• Incorporate gated CNNs (GLU activations) to capture sequential and hierarchical speech structures.
• Add an identity-mapping loss to preserve linguistic information and employ L1 loss for cycle and identity terms.
• Train with least-squares GAN objective to stabilize training.
• Represent source and target using 24 Mel-cepstral coefficients, log F0, and APs; convert MCEP domain and transform F0 accordingly.
• Use WORLD vocoder-based features and random-cropping of segments to increase batch diversity.

Experimental results

Research questions

• RQ1 Can a CycleGAN-based model learn source-to-target voice mappings without parallel data?
• RQ2 Does incorporating gated CNNs and an identity-mapping loss preserve linguistic information while reducing over-smoothing?
• RQ3 How does parallel-data-free CycleGAN-VC perform compared to a GMM-based VC when data conditions are constrained?
• RQ4 What objective measures (GV, MS) and subjective MOS scores indicate the quality of converted MCEPs?
• RQ5 Is CycleGAN-VC competitive under non-ideal data conditions with half the data and no parallelism?

Key findings

• CycleGAN-VC with GLU achieves MCEP sequences closest to target in GV and MS compared with ablations and GMM-VC baselines.
• Objective RMSE on log MS shows CycleGAN-VC w/ GLU outperforms CycleGAN-VC w/o GLU and non-GLU variants across speaker pairs.
• Subjective MOS indicates CycleGAN-VC outperforms the VCC 2016 baseline in naturalness under parallel-data-free conditions.
• CycleGAN-VC is comparable to a GMM-based method trained on parallel data with double the amount of data, despite non-parallel, smaller data.
• The approach reduces over-smoothing due to adversarial loss and benefits from GLU activations for modeling sequential structure.