[Paper Review] Neural Voice Cloning with a Few Samples

Sercan Ö. Arık, Jitong Chen+3 more|arXiv (Cornell University)|Feb 14, 2018

Speech Recognition and Synthesis43 references175 citations

TL;DR

The paper presents two methods to clone a speaker’s voice from few samples: speaker adaptation (fine-tuning a multi-speaker model) and speaker encoding (training a encoder to infer a speaker embedding from scant data). It evaluates naturalness and similarity, and introduces automated evaluation via speaker classification and verification, along with voice morphing via embedding manipulation.

ABSTRACT

Voice cloning is a highly desired feature for personalized speech interfaces. Neural network based speech synthesis has been shown to generate high quality speech for a large number of speakers. In this paper, we introduce a neural voice cloning system that takes a few audio samples as input. We study two approaches: speaker adaptation and speaker encoding. Speaker adaptation is based on fine-tuning a multi-speaker generative model with a few cloning samples. Speaker encoding is based on training a separate model to directly infer a new speaker embedding from cloning audios and to be used with a multi-speaker generative model. In terms of naturalness of the speech and its similarity to original speaker, both approaches can achieve good performance, even with very few cloning audios. While speaker adaptation can achieve better naturalness and similarity, the cloning time or required memory for the speaker encoding approach is significantly less, making it favorable for low-resource deployment.

Motivation & Objective

• Demonstrate the effectiveness of adapting a pre-trained multi-speaker TTS model to unseen speakers using few samples.
• Propose a speaker encoding approach that infers speaker embeddings from limited audio without fine-tuning the generator.
• Introduce automated evaluation methods based on speaker classification and speaker verification for cloning quality.
• Showcase voice morphing (gender and accent transformation) through embedding manipulations.
• Discuss trade-offs between cloning quality, cloning time, and resource requirements.

Proposed method

• Adopt a multi-speaker generative model (based on Deep Voice 3) parametrized by shared weights W and per-speaker embeddings e_s.
• Investigate two cloning strategies: (a) speaker adaptation by fine-tuning either the embedding alone or the whole model; (b) speaker encoding by training a separate encoder g(A_s; Θ) that predicts e_s from cloning audios.
• Train a speaker encoder with an L1 loss to match embeddings obtained from a pre-trained model, enabling zero-shot inference for unseen speakers.
• Use mel-spectrograms, prenets, temporal convolutions, and a multi-head self-attention mechanism to aggregate information from multiple cloning samples.
• Evaluate cloning quality with manual MOS and similarity tests, and with discriminative models (speaker classification and speaker verification).
• Demonstrate embedding-based voice morphing by manipulating latent embeddings to alter gender and accent.

Experimental results

Research questions

• RQ1 Can a pre-trained multi-speaker TTS model be quickly adapted to unseen speakers with only a few samples?
• RQ2 Is a dedicated speaker encoder capable of producing effective speaker embeddings without fine-tuning the generator?
• RQ3 How do cloning approaches compare in naturalness and speaker similarity, especially with limited data?
• RQ4 Can automated discriminative evaluations (speaker classification and verification) reliably assess cloning quality?
• RQ5 Can embeddings be manipulated to achieve controllable voice morphing (gender, accent)?

Key findings

• Both speaker adaptation and speaker encoding achieve good naturalness and similarity with few cloning audios.
• Whole-model adaptation generally yields better similarity with more cloning data, while embedding-only adaptation is faster to train.
• Speaker encoding offers substantially reduced cloning time and memory footprint, making it favorable for low-resource deployment.
• Automated evaluations using speaker classification and verification correlate with human judgments of cloning quality.
• Embedding manipulations can produce intelligible gender and accent transformations in cloned voices.