[Paper Review] AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss

Kaizhi Qian, Yang Zhang+3 more|arXiv (Cornell University)|May 14, 2019

Speech Recognition and Synthesis32 references195 citations

TL;DR

AutoVC performs zero-shot voice style transfer using a bottlenecked autoencoder trained solely on self-reconstruction, enabling non-parallel many-to-many voice conversion.

ABSTRACT

Non-parallel many-to-many voice conversion, as well as zero-shot voice conversion, remain under-explored areas. Deep style transfer algorithms, such as generative adversarial networks (GAN) and conditional variational autoencoder (CVAE), are being applied as new solutions in this field. However, GAN training is sophisticated and difficult, and there is no strong evidence that its generated speech is of good perceptual quality. On the other hand, CVAE training is simple but does not come with the distribution-matching property of a GAN. In this paper, we propose a new style transfer scheme that involves only an autoencoder with a carefully designed bottleneck. We formally show that this scheme can achieve distribution-matching style transfer by training only on a self-reconstruction loss. Based on this scheme, we proposed AUTOVC, which achieves state-of-the-art results in many-to-many voice conversion with non-parallel data, and which is the first to perform zero-shot voice conversion.

Motivation & Objective

• Motivate non-parallel, many-to-many voice conversion without parallel data.
• Develop a simple autoencoder-based style transfer that matches target distributions.
• Enable zero-shot conversion to unseen speakers using a pre-trained speaker embedding.
• Disentangle content from speaker information via a carefully designed bottleneck.
• Provide theoretical justification that the autoencoder achieves distribution matching.

Proposed method

• Propose a three-module framework: content encoder, speaker encoder, and decoder.
• Train with self-reconstruction loss and a content consistency loss to enforce disentanglement.
• Use a carefully designed bottleneck via temporal downsampling and channel reduction to separate content and speaker information.
• Pre-train a speaker encoder with GE2E loss to enable unseen speaker embeddings for zero-shot conversion.
• Decode using a spectrogram-based autoencoder with a post-net to refine spectrogram details.
• Incorporate a WaveNet vocoder to synthesize the final waveform from the converted spectrogram.

Experimental results

Research questions

• RQ1 Can an autoencoder trained only on self-reconstruction achieve distribution-matching style transfer for voice conversion?
• RQ2 Can the bottleneck be tuned to disentangle speaker information from content to enable zero-shot conversion?
• RQ3 Does the proposed AutoVC framework perform well on non-parallel, many-to-many voice conversion, including unseen speakers?
• RQ4 Is a pre-trained speaker embedding sufficient to support zero-shot conversion without parallel data?

Key findings

• AutoVC achieves state-of-the-art performance on traditional non-parallel many-to-many voice conversion with seen speakers.
• AutoVC enables zero-shot voice conversion using unseen speakers with decent performance.
• Subjective MOS scores for AutoVC surpass baseline non-parallel methods and approach parallel/conversion quality.
• A well-tuned bottleneck balances reconstruction quality and speaker disentanglement, enabling effective conversion.
• AutoVC-one-hot experiments show the speaker embedding contributes less to gains than the bottleneck design.
• Table 1 shows a trade-off between reconstruction quality and disentanglement across narrow, AutoVC, and wide bottlenecks.