[Paper Review] Robust One-Shot Singing Voice Conversion

Naoya Takahashi, Mayank Singh+1 more|arXiv (Cornell University)|Oct 20, 2022

Music and Audio Processing4 citations

TL;DR

This paper proposes Robust One-Shot Singing Voice Conversion (ROSVC), a novel framework that enables high-fidelity, pitch-accurate voice conversion for any unseen singer using less than 10 seconds of reference audio, even when the input is distorted by reverb and accompaniment music. It combines a GAN-based one-shot SVC model with pitch distribution matching and AdaIN-skip conditioning, a two-stage training strategy (Robustify) to enhance robustness against distortions, and a hierarchical diffusion-based neural vocoder for improved voice quality and pitch stability, outperforming state-of-the-art methods in both subjective and objective evaluations.

ABSTRACT

Recent progress in deep generative models has improved the quality of voice conversion in the speech domain. However, high-quality singing voice conversion (SVC) of unseen singers remains challenging due to the wider variety of musical expressions in pitch, loudness, and pronunciation. Moreover, singing voices are often recorded with reverb and accompaniment music, which make SVC even more challenging. In this work, we present a robust one-shot SVC (ROSVC) that performs any-to-any SVC robustly even on such distorted singing voices. To this end, we first propose a one-shot SVC model based on generative adversarial networks that generalizes to unseen singers via partial domain conditioning and learns to accurately recover the target pitch via pitch distribution matching and AdaIN-skip conditioning. We then propose a two-stage training method called Robustify that train the one-shot SVC model in the first stage on clean data to ensure high-quality conversion, and introduces enhancement modules to the encoders of the model in the second stage to enhance the feature extraction from distorted singing voices. To further improve the voice quality and pitch reconstruction accuracy, we finally propose a hierarchical diffusion model for singing voice neural vocoders. Experimental results show that the proposed method outperforms state-of-the-art one-shot SVC baselines for both seen and unseen singers and significantly improves the robustness against distortions.

Motivation & Objective

• To address the challenge of high-quality singing voice conversion (SVC) for unseen singers in realistic, distorted conditions such as reverb and background music.
• To improve robustness of one-shot SVC models against acoustic distortions without relying on pre-processing like dereverberation or source separation.
• To achieve accurate pitch recovery and natural voice quality in one-shot SVC, even when the target singer is not in the training data.
• To develop a neural vocoder that enhances perceptual quality and pitch stability in converted singing voices.

Proposed method

• Proposes a one-shot SVC model based on generative adversarial networks (GANs) with partial domain conditioning and pitch distribution matching to generalize to unseen singers.
• Introduces AdaIN-skip conditioning to improve pitch reconstruction accuracy by preserving pitch-related features during the conversion process.
• Develops a two-stage training framework called Robustify: first pre-trains on clean data for high-quality conversion, then fine-tunes with enhancement modules in the encoder to improve robustness to reverb and music interference.
• Designs a hierarchical diffusion model-based neural vocoder (HPG) that uses multiple diffusion models at different sampling rates to generate high-fidelity singing voices with improved pitch stability.
• Employs a multi-scale conditioning strategy where lower sampling rate waveforms are used directly as conditioning to guide low-frequency component generation.
• Validates the method on NUS48E, NHSS, and MUSDB18 datasets, comparing against five state-of-the-art one-shot VC baselines.

Experimental results

Research questions

• RQ1 Can a one-shot SVC model achieve high-quality voice conversion for unseen singers while maintaining accurate pitch and naturalness?
• RQ2 How can a one-shot SVC model be made robust to real-world distortions such as reverb and background music without relying on pre-processing?
• RQ3 Does a hierarchical diffusion-based neural vocoder improve the perceptual quality and pitch accuracy of converted singing voices compared to autoregressive or GAN-based vocoders?
• RQ4 To what extent does the two-stage training strategy (Robustify) enhance feature extraction robustness from distorted inputs?
• RQ5 Can direct conditioning on lower sampling rate waveforms improve the fidelity and stability of the generated singing voice?

Key findings

• The proposed ROSVC model achieves a MOS score of 4.07, significantly outperforming PriorGrad (3.70) and PWG (2.81), demonstrating superior perceptual quality.
• In preference tests, 85.3% of raters preferred the HPG-2 model over PriorGrad, confirming its naturalness and audio quality.
• The hierarchical diffusion vocoder (HPG-2) achieves a PMAE of 1.82, lower than PriorGrad (1.80) and PWG (3.12), indicating better pitch accuracy.
• HPG-3, a 3-stage hierarchical model, further reduces PMAE to 1.67 and MCD to 8.12, showing improved performance with only a 30% increase in computational cost.
• Ablation studies confirm that conditioning on low-rate waveforms ($x_0^2$) is critical for generating low-frequency components, while mel-spectrograms alone are insufficient for full-spectrum fidelity.
• The Robustify training strategy significantly improves robustness to reverb and music interference, enabling stable performance on distorted inputs where baselines fail.