[Paper Review] Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion

Joan Serrà, Santiago Pascual+1 more|arXiv (Cornell University)|Jun 3, 2019

Speech Recognition and SynthesisComputer Science50 references35 citations

TL;DR

Blow is a single-scale normalizing flow with hypernetwork conditioning that performs many-to-many non-parallel voice conversion on raw audio, achieving competitive objective and subjective results. It uses forward-backward conversion, shared embeddings, and data augmentation to improve performance.

ABSTRACT

End-to-end models for raw audio generation are a challenge, specially if they have to work with non-parallel data, which is a desirable setup in many situations. Voice conversion, in which a model has to impersonate a speaker in a recording, is one of those situations. In this paper, we propose Blow, a single-scale normalizing flow using hypernetwork conditioning to perform many-to-many voice conversion between raw audio. Blow is trained end-to-end, with non-parallel data, on a frame-by-frame basis using a single speaker identifier. We show that Blow compares favorably to existing flow-based architectures and other competitive baselines, obtaining equal or better performance in both objective and subjective evaluations. We further assess the impact of its main components with an ablation study, and quantify a number of properties such as the necessary amount of training data or the preference for source or target speakers.

Motivation & Objective

• Address non-parallel, many-to-many voice conversion on raw audio.
• Develop a single-scale flow architecture with hyperconditioning for speaker identity.
• Evaluate Blaze against flow-based baselines and non-flow voice-conversion systems.
• Perform ablation studies to quantify the impact of architecture and data-augmentation components.

Proposed method

• Use a single-scale Glow-inspired flow with 8 blocks and 12 flows per block (8x12).
• Employ forward-backward conversion where source x(S) maps to latent z under source conditioning and z maps to x(T) under target conditioning.
• Implement hyperconditioning by conditioning the first layer of the coupling network with weights generated by a hypernetwork from a shared speaker embedding.
• Share a single speaker embedding across all coupling networks and all steps to constrain conditioning.
• Augment raw audio frames with temporal jitter, pre-/de-emphasis, amplitude scaling, and sign flipping.
• Train to maximize exact log-likelihood with an isotropic Gaussian prior on latent z.

Experimental results

Research questions

• RQ1 Can a single-scale hyperconditioned flow achieve competitive non-parallel, many-to-many voice conversion on raw audio?
• RQ2 What is the impact of forward-backward conversion and shared embeddings on conversion quality and likelihood?
• RQ3 How do data augmentation strategies affect performance and robustness?
• RQ4 What is the relative importance of each architectural component (hyperconditioning, shared embeddings, single-scale structure) on objective and subjective metrics?

Key findings

• Blow achieves higher likelihood (L) than Glow and Glow-WaveNet in objective evaluations.
• Blow attains higher spoofing accuracy than VQ-VAE, indicating better speaker-identity transfer resistance.
• Subjective results show Blow has comparable naturalness to StarGAN and higher similarity to the target than StarGAN or VQ-VAE.
• Ablation studies show single-scale structure as the most critical component for performance.
• Data augmentation significantly improves both objective and spoofing metrics.
• Conversion performance depends strongly on the target speaker identity, with source identity having less influence.