[Paper Review] Natural language guidance of high-fidelity text-to-speech with synthetic annotations

Dan Lyth, Simon King|arXiv (Cornell University)|Feb 2, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes a scalable, fully automatic method for generating high-fidelity text-to-speech using natural language conditioning by synthesizing speech attributes (e.g., accent, pitch, speaking rate, recording conditions) via self-supervised labeling on a 45k-hour dataset. The approach achieves state-of-the-art audio quality and naturalness—outperforming prior work like Audiobox—using only 1% high-fidelity data and a modern audio codec, all with a single model and zero-shot control via prompts.

ABSTRACT

Text-to-speech models trained on large-scale datasets have demonstrated impressive in-context learning capabilities and naturalness. However, control of speaker identity and style in these models typically requires conditioning on reference speech recordings, limiting creative applications. Alternatively, natural language prompting of speaker identity and style has demonstrated promising results and provides an intuitive method of control. However, reliance on human-labeled descriptions prevents scaling to large datasets. Our work bridges the gap between these two approaches. We propose a scalable method for labeling various aspects of speaker identity, style, and recording conditions. We then apply this method to a 45k hour dataset, which we use to train a speech language model. Furthermore, we propose simple methods for increasing audio fidelity, significantly outperforming recent work despite relying entirely on found data. Our results demonstrate high-fidelity speech generation in a diverse range of accents, prosodic styles, channel conditions, and acoustic conditions, all accomplished with a single model and intuitive natural language conditioning. Audio samples can be heard at https://text-description-to-speech.com/.

Motivation & Objective

• To overcome the scalability bottleneck of human-annotated speech descriptions for natural language-controlled TTS.
• To enable high-fidelity, diverse speech generation across accents, prosody, and recording conditions using only natural language prompts.
• To train a unified speech language model that generalizes to unseen combinations of speaker identity, style, and channel conditions.
• To achieve high audio quality with minimal high-fidelity data by leveraging state-of-the-art audio codecs.
• To demonstrate that automatic labeling can match or exceed the performance of human-annotated baselines in TTS control and quality.

Proposed method

• Automatically extract and label speech attributes (gender, accent, pitch, speaking rate, channel conditions) using a combination of computational metrics and self-supervised modeling on a 45k-hour dataset.
• Use a large-scale speech language model fine-tuned on the automatically annotated data to generate speech conditioned on natural language descriptions.
• Apply the DAC audio codec to enhance audio fidelity, even when training data contains only 1% high-fidelity samples.
• Leverage a small, high-quality subset (LibriTTS-R, ~500 hours) to guide the model toward high-fidelity output despite the majority of data being low-fidelity.
• Generate diverse speech samples by prompting the model with natural language descriptions that control multiple attributes independently.
• Use loudness normalization and silence trimming for fair subjective evaluation, ensuring consistent comparison with ground truth and baselines.

Figure 1 : Overview of the model architecture

Experimental results

Research questions

• RQ1 Can fully automatic, synthetic annotation of speech attributes enable scalable, high-fidelity text-to-speech with natural language control?
• RQ2 To what extent can a single model generalize to unseen combinations of speaker identity, prosody, and recording conditions using only natural language prompts?
• RQ3 Can high audio fidelity be achieved with minimal high-fidelity data by combining synthetic annotations and advanced audio codecs?
• RQ4 How does the performance of a model trained on automatically labeled data compare to models trained on human-annotated data in terms of naturalness and alignment with descriptions?
• RQ5 Does the use of a modern audio codec (DAC) significantly improve audio quality when combined with synthetic labeling and limited high-fidelity data?

Key findings

• The model achieves a MOS of 3.92, significantly outperforming Audiobox (2.79) and even surpassing ground truth (3.67) in subjective naturalness evaluation.
• In relevance to description alignment, the model scores 3.88, outperforming Audiobox (3.19) and ground truth (3.62), indicating better adherence to prompt instructions.
• The model achieves a PESQ score of 3.84, close to ground truth (4.15), and STOI of 0.996, indicating high speech intelligibility and quality.
• Despite training on only ~500 hours of high-fidelity data (1% of total), the model produces audio with perceptual quality exceeding that of Audiobox, which uses more data and human annotations.
• Objective metrics show that the model’s synthesized labels (e.g., pitch, speaking rate) correlate strongly with description labels, with 95% confidence intervals indicating high reliability.
• The model generalizes to unseen combinations of attributes, such as non-native accents and noisy recording conditions, all controlled via a single natural language prompt.

Figure 2 : Estimated SNR across the MLS dataset and the discrete bin boundaries used to create keywords.