[Paper Review] Improving Zero-shot Voice Style Transfer via Disentangled Representation Learning

Siyang Yuan, Pengyu Cheng+4 more|arXiv (Cornell University)|Mar 17, 2021

Speech Recognition and SynthesisComputer Science53 references27 citations

TL;DR

IDE-VC proposes information-theoretic disentangled embeddings to achieve improved zero-shot and many-to-many voice style transfer, outperforming baselines on VCTK.

ABSTRACT

Voice style transfer, also called voice conversion, seeks to modify one speaker's voice to generate speech as if it came from another (target) speaker. Previous works have made progress on voice conversion with parallel training data and pre-known speakers. However, zero-shot voice style transfer, which learns from non-parallel data and generates voices for previously unseen speakers, remains a challenging problem. We propose a novel zero-shot voice transfer method via disentangled representation learning. The proposed method first encodes speaker-related style and voice content of each input voice into separated low-dimensional embedding spaces, and then transfers to a new voice by combining the source content embedding and target style embedding through a decoder. With information-theoretic guidance, the style and content embedding spaces are representative and (ideally) independent of each other. On real-world VCTK datasets, our method outperforms other baselines and obtains state-of-the-art results in terms of transfer accuracy and voice naturalness for voice style transfer experiments under both many-to-many and zero-shot setups.

Motivation & Objective

• Motivate zero-shot voice style transfer from non-parallel data and many-to-many scenarios.
• Propose a disentangled representation learning framework to separate style and content in embeddings.
• Incorporate mutual information bounds to encourage independence between style and content.
• Leverage speaker identity as supervision to stabilize learned style representations.
• Deliver improved transfer quality and naturalness over prior encoder-decoder VST models.

Proposed method

• Encode input speech x into a style embedding s = E_s(x) and a content embedding c = E_c(x).
• Decode via D(s, c) to reconstruct or transfer to a target style by combining source content with target style.
• Minimize a mutual information based objective to disentangle s and c, using upper/lower bounds for I(s; c), I(x; c|s), and I(u; s).
• Provide two new multi-group MI bounds (Theorem 3.1 and Theorem 3.2) to reinforce style consistency and content preservation.
• Approximate unknown conditional p(s|c) with a neural q_theta(s|c) to bound I(s; c) and iteratively train E_s, E_c, D, and q_theta.
• Utilize a pre-trained GE2E-based style encoder initialization and WaveNet vocoder for waveform synthesis.

Experimental results

Research questions

• RQ1 Can zero-shot voice style transfer be reliably achieved with disentangled style and content representations under non-parallel data?
• RQ2 Do mutual-information-based disentanglement bounds improve transfer fidelity and naturalness compared to prior encoder-decoder approaches?
• RQ3 How does IDE-VC perform in many-to-many and zero-shot VST settings on VCTK relative to baselines?
• RQ4 Do the learned style embeddings align with speaker identity while content embeddings remain speaker-invariant?
• RQ5 Are the proposed MI bounds practically estimable and beneficial when training end-to-end models?

Key findings

• IDE-VC outperforms baselines (StarGAN, Blow, AUTOVC, AdaIN-VC) on many-to-many VST in both objective and subjective metrics.
• Zero-shot VST: IDE-VC surpasses AUTOVC and AdaIN-VC across all reported metrics.
• Style embeddings cluster by speaker while content embeddings are speaker-invariant (t-SNE visualization).
• Content embeddings achieve lower speaker-identity predictability than baselines, indicating better disentanglement.
• Ablation studies show removing the MI-based terms degrades transfer quality, confirming the importance of disentanglement losses.