[Paper Review] Towards Fine-Grained Prosody Control for Voice Conversion

Zheng Lian, Zhengqi Wen|arXiv (Cornell University)|Oct 24, 2019

Speech Recognition and Synthesis27 references4 citations

TL;DR

This paper proposes a non-parallel voice conversion framework that uses unsupervised prosody embeddings to enable fine-grained prosody control, improving speech quality and speaker similarity—especially in singing conditions—by replacing pitch-based prosody control with learned embeddings and employing the real-time LPCNet vocoder for high-fidelity synthesis.

ABSTRACT

In a typical voice conversion system, prior works utilize various acoustic features (e.g., the pitch, voiced/unvoiced flag, aperiodicity) of the source speech to control the prosody of generated waveform. However, the prosody is related with many factors, such as the intonation, stress and rhythm. It is a challenging task to perfectly describe the prosody through acoustic features. To deal with this problem, we propose prosody embeddings to model prosody. These embeddings are learned from the source speech in an unsupervised manner. We conduct experiments on our Mandarin corpus recoded by professional speakers. Experimental results demonstrate that the proposed method enables fine-grained control of the prosody. In challenging situations (such as the source speech is a singing song), our proposed method can also achieve promising results.

Motivation & Objective

• To address the limitations of pitch-based prosody control in voice conversion, which fails to capture complex prosodic factors like intonation, stress, and rhythm.
• To improve speech quality and speaker similarity in non-parallel voice conversion by replacing handcrafted acoustic features with unsupervised prosody embeddings.
• To enable real-time, high-fidelity speech synthesis using the LPCNet neural vocoder, which offers better spectral control and efficiency than traditional parametric vocoders.
• To evaluate the proposed method on both standard TTS-style speech and singing conditions, demonstrating robustness across diverse prosodic patterns.

Proposed method

• Prosody embeddings are learned from source speech in an unsupervised manner using a neural network, capturing complex prosodic characteristics without explicit annotations.
• The framework uses phonetic posteriorgrams (PPGs) as linguistic input and maps them to target speaker acoustic features using a CBHG-based conversion model.
• The prosody embeddings are concatenated with PPGs and used as conditional inputs to guide the acoustic feature generation process.
• Speech waveforms are synthesized using LPCNet, a real-time neural vocoder that predicts excitation signals based on linear prediction and RNNs, improving spectral fidelity.
• The system is trained end-to-end with Adam optimizer, batch size 32, and gradient clipping, using 100k steps for convergence.
• A baseline system using pitch and vuv for prosody control is compared against the proposed method and a PPG-based baseline (C1) using WORLD vocoder.

Experimental results

Research questions

• RQ1 Can unsupervised prosody embeddings effectively model complex prosodic factors such as intonation, stress, and rhythm beyond pitch?
• RQ2 Does replacing pitch-based prosody control with learned embeddings improve speech quality and speaker similarity in non-parallel voice conversion?
• RQ3 Can the proposed framework achieve high-quality, real-time speech synthesis using LPCNet, especially in challenging singing conditions?
• RQ4 How does the proposed method compare to existing PPG-based non-parallel VC systems in terms of subjective quality and speaker similarity?

Key findings

• The proposed method significantly outperforms the pitch-based baseline in singing-to-speech conversion (SONG-to-TS pair), demonstrating superior prosody control.
• Subjective MOS results show that the proposed method achieves higher speech naturalness than the baseline and C1 system on both CMU-ARCTIC and Mandarin datasets.
• The Same/Different paradigm results indicate that the proposed method achieves higher speaker similarity than the baseline, particularly in non-speech prosodic conditions.
• LPCNet outperforms the WORLD vocoder in both speech quality and speaker similarity, confirming its effectiveness for real-time, high-fidelity synthesis.
• The method achieves promising results in singing conditions, where pitch-based control fails due to pitch detection errors and insufficient prosodic modeling.
• The use of 1D prosody embeddings with a small bottleneck size enables effective, fine-grained prosody transfer without requiring parallel data or explicit annotations.