[Paper Review] Phonetic Posteriorgrams based Many-to-Many Singing Voice Conversion via Adversarial Training

Haohan Guo, Heng Lu+6 more|arXiv (Cornell University)|Dec 3, 2020

Music and Audio Processing26 references10 citations

TL;DR

This paper proposes an end-to-end adversarial singing voice conversion (EA-SVC) system that generates high-fidelity singing waveforms from phonetic posteriorgrams (PPG), F0, and speaker embeddings using a MelGAN-based generator with CNN-BLSTM encoders and adversarial training. The key contributions include speaker embedding decomposition into timbre clusters, a feature disentanglement discriminator to remove pitch/timbre leakage from PPG, and a two-stage training strategy, resulting in superior singing quality and singer similarity over cascade and WaveNet-based methods.

ABSTRACT

This paper describes an end-to-end adversarial singing voice conversion (EA-SVC) approach. It can directly generate arbitrary singing waveform by given phonetic posteriorgram (PPG) representing content, F0 representing pitch, and speaker embedding representing timbre, respectively. Proposed system is composed of three modules: generator $G$, the audio generation discriminator $D_{A}$, and the feature disentanglement discriminator $D_F$. The generator $G$ encodes the features in parallel and inversely transforms them into the target waveform. In order to make timbre conversion more stable and controllable, speaker embedding is further decomposed to the weighted sum of a group of trainable vectors representing different timbre clusters. Further, to realize more robust and accurate singing conversion, disentanglement discriminator $D_F$ is proposed to remove pitch and timbre related information that remains in the encoded PPG. Finally, a two-stage training is conducted to keep a stable and effective adversarial training process. Subjective evaluation results demonstrate the effectiveness of our proposed methods. Proposed system outperforms conventional cascade approach and the WaveNet based end-to-end approach in terms of both singing quality and singer similarity. Further objective analysis reveals that the model trained with the proposed two-stage training strategy can produce a smoother and sharper formant which leads to higher audio quality.

Motivation & Objective

• To develop an end-to-end singing voice conversion system that enables arbitrary timbre transfer with high audio quality and speaker similarity.
• To address the challenge of interdependent content, pitch, and timbre representations in singing voice conversion by ensuring feature disentanglement.
• To improve audio fidelity and stability by integrating adversarial training with a two-stage training strategy and multi-resolution STFT loss.
• To enable controllable timbre transfer and pitch manipulation through disentangled speaker embedding decomposition.
• To outperform conventional cascade and WaveNet-based end-to-end approaches in both subjective and objective evaluations.

Proposed method

• A MelGAN-based generator maps PPG, F0, and speaker embedding to waveform, using parallel CNN-BLSTM encoders to extract sequential context from PPG and F0.
• Speaker embedding is decomposed into a weighted sum of trainable vectors representing distinct timbre/speaker clusters to improve timbre representation stability and controllability.
• Two discriminators are employed: $D_A$ for audio generation to improve waveform realism, and $D_F$ to disentangle pitch and timbre information from the encoded PPG.
• A two-stage training strategy is used: first pre-training with multi-resolution STFT (MR-STFT) loss, then joint training with MR-STFT and adversarial loss to stabilize GAN training.
• The feature disentanglement discriminator $D_F$ is trained to predict F0 and speaker embedding from encoded PPG, forcing the PPG representation to be free of pitch and timbre information.
• The system is trained end-to-end with adversarial loss and reconstruction loss, enabling direct waveform generation from content, pitch, and timbre features.

Experimental results

Research questions

• RQ1 Can a single end-to-end model achieve high-quality singing voice conversion with arbitrary timbre transfer and stable pitch control?
• RQ2 How effective is speaker embedding decomposition into timbre clusters in improving timbre representation and conversion controllability?
• RQ3 Does the inclusion of a feature disentanglement discriminator ($D_F$) reduce pitch and timbre leakage in the PPG representation and improve audio quality?
• RQ4 To what extent does the two-stage training strategy enhance the stability and performance of adversarial training in singing voice conversion?
• RQ5 How does the proposed method compare to cascade and WaveNet-based end-to-end approaches in terms of singing quality and singer similarity?

Key findings

• The proposed EA-SVC model achieves a mean opinion score (MOS) of 3.78 ± 0.11 for singing quality and 3.60 ± 0.13 for singer similarity, outperforming both the cascade approach and WaveNet-based end-to-end method.
• The ablation study shows that removing both speaker embedding decomposition and $D_F$ results in the worst performance (3.26 ± 0.11 for quality), confirming the necessity of both components.
• The model with speaker embedding decomposition alone improves MOS to 3.52 ± 0.12 (quality) and 3.55 ± 0.12 (similarity), indicating enhanced timbre representation robustness.
• The feature disentanglement discriminator $D_F$ significantly reduces pitch glitches and formant fuzziness, leading to clearer and sharper formants in spectrograms.
• The two-stage training strategy produces smoother and sharper formants compared to models trained only with adversarial loss, as shown in spectrogram comparisons.
• Timbre transfer experiments confirm that decomposition enables linear, controllable interpolation between timbres, while the non-decomposed model fails to maintain consistent spectral trends.