[Paper Review] Data Efficient Voice Cloning from Noisy Samples with Domain Adversarial Training

Jian Cong, Shan Yang+3 more|arXiv (Cornell University)|Aug 10, 2020

Speech Recognition and Synthesis28 references4 citations

TL;DR

This paper proposes domain adversarial training (DAT) to enable data-efficient voice cloning from noisy target speaker samples in both speaker adaptation and speaker encoding settings. By disentangling noise from speaker identity in latent representations, the method achieves superior speech quality and speaker similarity compared to baseline models and those using external speech enhancement, even with only a few noisy samples.

ABSTRACT

Data efficient voice cloning aims at synthesizing target speaker's voice with only a few enrollment samples at hand. To this end, speaker adaptation and speaker encoding are two typical methods based on base model trained from multiple speakers. The former uses a small set of target speaker data to transfer the multi-speaker model to target speaker's voice through direct model update, while in the latter, only a few seconds of target speaker's audio directly goes through an extra speaker encoding model along with the multi-speaker model to synthesize target speaker's voice without model update. Nevertheless, the two methods need clean target speaker data. However, the samples provided by user may inevitably contain acoustic noise in real applications. It's still challenging to generating target voice with noisy data. In this paper, we study the data efficient voice cloning problem from noisy samples under the sequence-to-sequence based TTS paradigm. Specifically, we introduce domain adversarial training (DAT) to speaker adaptation and speaker encoding, which aims to disentangle noise from speech-noise mixture. Experiments show that for both speaker adaptation and encoding, the proposed approaches can consistently synthesize clean speech from noisy speaker samples, apparently outperforming the method adopting state-of-the-art speech enhancement module.

Motivation & Objective

• To address the challenge of data-efficient voice cloning when target speaker samples are noisy, which degrades performance in existing seq2seq TTS systems.
• To improve speaker adaptation and speaker encoding methods by making them robust to noise without requiring clean data or external enhancement modules.
• To disentangle noise from speaker identity in latent representations using domain adversarial training, ensuring stable and high-quality voice synthesis.
• To evaluate the effectiveness of the proposed method across both few-shot adaptation and one-shot encoding scenarios under noisy conditions.

Proposed method

• Introduce domain adversarial training (DAT) to the decoder of a sequence-to-sequence TTS model, training a domain classifier to distinguish between clean and noisy speech inputs.
• Modify the speaker adaptation framework by adding a domain classifier and noise control tags to the decoder, enabling the model to learn noise-invariant speaker representations during fine-tuning.
• Extend the speaker encoding framework with a domain classifier and an external speaker encoder, ensuring that speaker embeddings remain robust to noise in the input audio.
• Use gradient reversal during backpropagation to enforce domain invariance, encouraging the model to learn speaker-specific features independent of noise.
• Train the TTS model on a mixed dataset of clean and noisy speaker samples, with the domain classifier penalizing the model for learning noise-dependent features.
• Apply t-SNE visualization to demonstrate that the proposed method clusters noisy and clean samples of the same speaker closely, indicating effective noise disentanglement.

Experimental results

Research questions

• RQ1 Can domain adversarial training improve speaker adaptation performance when fine-tuning with only a few noisy target speaker samples?
• RQ2 Can speaker encoding maintain high speaker similarity and naturalness when using a single noisy sample as input?
• RQ3 Does the proposed method outperform state-of-the-art speech enhancement modules when used as a preprocessing step for voice cloning?
• RQ4 How effectively does domain adversarial training disentangle noise from speaker identity in latent representations?
• RQ5 Is the proposed method robust across different speaker genders and varying noise levels in test conditions?

Key findings

• The proposed DAT-based speaker adaptation model achieved a mean opinion score (MOS) of 3.65 for similarity and 3.36 for naturalness on noisy female samples, outperforming the baseline which failed to converge on such data.
• For one-shot speaker encoding, the proposed method achieved a similarity score of 0.88 on noisy male samples, compared to 0.76 for the baseline using de-noised inputs, indicating superior robustness.
• The model using domain adversarial training maintained consistent performance across clean, noisy, and de-noised test sets, while the baseline model failed to adapt on noisy samples in both adaptation and encoding settings.
• t-SNE visualization confirmed that the proposed method clusters noisy and clean samples of the same speaker closely, while the baseline model showed distinct clusters, indicating noise-influenced embeddings.
• The proposed method achieved lower MCD (Mel-cepstral distortion) than the de-noising baseline, with 4.18 for female adaptation on noisy data, compared to 4.72 for the de-noising approach.
• In speaker encoding, the proposed method achieved a MOS of 3.34 for similarity on noisy male samples, significantly outperforming the baseline's 2.96 on de-noised data.