[Paper Review] SpeechX: Neural Codec Language Model as a Versatile Speech Transformer

Xiaofei Wang, Manthan Thakker+8 more|arXiv (Cornell University)|Aug 14, 2023

Speech Recognition and SynthesisComputer Science3 citations

TL;DR

SpeechX is a unified neural codec language model that enables versatile speech generation and transformation via multi-task learning with task-specific prompting, achieving state-of-the-art or competitive performance across zero-shot TTS, noise suppression, target speaker extraction, speech removal, and speech editing—both in clean and noisy conditions—using a single unified architecture.

ABSTRACT

Recent advancements in generative speech models based on audio-text prompts have enabled remarkable innovations like high-quality zero-shot text-to-speech. However, existing models still face limitations in handling diverse audio-text speech generation tasks involving transforming input speech and processing audio captured in adverse acoustic conditions. This paper introduces SpeechX, a versatile speech generation model capable of zero-shot TTS and various speech transformation tasks, dealing with both clean and noisy signals. SpeechX combines neural codec language modeling with multi-task learning using task-dependent prompting, enabling unified and extensible modeling and providing a consistent way for leveraging textual input in speech enhancement and transformation tasks. Experimental results show SpeechX's efficacy in various tasks, including zero-shot TTS, noise suppression, target speaker extraction, speech removal, and speech editing with or without background noise, achieving comparable or superior performance to specialized models across tasks. See https://aka.ms/speechx for demo samples.

Motivation & Objective

• Address the lack of unified generative models that handle diverse audio-text speech tasks, including transformation of noisy and multi-talker speech.
• Overcome limitations of prior models that cannot perform speech editing on noisy signals or handle target speaker extraction without specialized architectures.
• Enable robust, zero-shot speech generation and transformation using a single, extensible model architecture with consistent conditioning on text and acoustic prompts.
• Unify multiple speech tasks—TTS, denoising, speaker extraction, removal, editing—under a single neural codec language modeling framework with task-specific tokens.
• Improve model robustness to acoustic distortions by training on diverse speech conditions while maintaining high-quality output across tasks.

Proposed method

• Employ a neural codec language model to generate discrete acoustic tokens conditioned on both text and acoustic prompts, enabling end-to-end speech generation and transformation.
• Introduce task-specific conditioning tokens (e.g., 'TTS', 'denoise', 'extract') to unify diverse tasks within a single model architecture using multi-task learning.
• Use text input as an optional but informative prompt to guide speech enhancement and transformation, improving disentanglement and intelligibility.
• Train the model end-to-end on a mixture of tasks, including zero-shot TTS, speech editing, noise suppression, speech removal, and target speaker extraction, with shared acoustic and text encoders.
• Leverage the EnCodec model for acoustic tokenization, though the performance is constrained by codec-induced degradation, especially in PESQ scores.
• Apply a unified inference mechanism where the model generates acoustic tokens conditioned on the input audio, text (if available), and task-specific tokens.

Experimental results

Research questions

• RQ1 Can a single unified generative model effectively handle diverse audio-text-based speech tasks, including zero-shot TTS and speech transformation in noisy environments?
• RQ2 How does incorporating text input as a prompt improve performance in speech enhancement and transformation tasks like noise suppression and target speaker extraction?
• RQ3 To what extent does multi-task training improve robustness and generalization across different speech generation and transformation tasks?
• RQ4 What is the impact of using a neural codec on the performance of downstream speech generation and transformation tasks, especially in terms of quality metrics like PESQ and DNSMOS?
• RQ5 Can a unified model achieve competitive performance in speech editing on noisy signals, where prior models are limited to clean inputs?

Key findings

• SpeechX achieves competitive or superior performance compared to specialized models across all evaluated tasks, including zero-shot TTS, noise suppression, target speaker extraction, speech removal, and speech editing.
• The inclusion of text input significantly improves DNSMOS and WER scores in target speaker extraction, indicating better disentanglement of the target speaker’s voice from interfering talkers.
• Training on speech editing tasks improves zero-shot TTS performance, suggesting beneficial data distribution shifts from task diversity.
• Adding noise suppression and speech removal tasks during training enhances robustness to noise in speech editing, albeit with a slight trade-off in clean speech editing quality.
• The use of EnCodec for acoustic tokenization leads to a notable PESQ degradation (from 4.64 to 2.69 on clean speech), indicating a performance bottleneck due to codec mismatch with evaluation metrics.
• SpeechX maintains strong performance in noise suppression and speech removal even when target speaker extraction is included in training, showing no degradation in these tasks.