[Paper Review] Synthesizing Personalized Non-speech Vocalization from Discrete Speech Representations

Chin-Cheng Hsu|arXiv (Cornell University)|Jun 25, 2022

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes a text-to-speech framework for synthesizing personalized non-speech vocalizations (NSVs) using discrete speech units from HUBERT. By treating NSVs as a text-to-speech task with pseudo-phonemes derived from HUBERT units, the model successfully synthesizes NSVs with controllable speaker timbre despite few-shot training and heterogeneous recording conditions, demonstrating viability for voice conversion and speaker modeling from NSV data.

ABSTRACT

We formulated non-speech vocalization (NSV) modeling as a text-to-speech task and verified its viability. Specifically, we evaluated the phonetic expressivity of HUBERT speech units on NSVs and verified our model's ability to control over speaker timbre even though the training data is speaker few-shot. In addition, we substantiated that the heterogeneity in recording conditions is the major obstacle for NSV modeling. Finally, we discussed five improvements over our method for future research. Audio samples of synthesized NSVs are available on our demo page: https://resemble-ai.github.io/reLaugh.

Motivation & Objective

• Address the challenge of non-speech vocalization (NSV) modeling due to data sparsity, spontaneity, and lack of standardized forms.
• Investigate whether discrete speech representations (HUBERT units) can serve as a meaningful text-like input for NSV synthesis.
• Evaluate the feasibility of controlling speaker timbre in NSV synthesis with limited (few-shot) data per speaker.
• Assess the impact of heterogeneous recording conditions on speaker embedding space and model generalization.
• Explore the potential of NSV-based voice conversion and speaker modeling using end-to-end TTS frameworks.

Proposed method

• Transcribe NSV audio into pseudo-phonemes using a pre-trained HUBERT model and K-means clustering (100 clusters), followed by run-length encoding to produce a sequence of discrete units.
• Use a modified FastSpeech2 architecture with dilated convolutions instead of transformers, a duration predictor, and a parallel pitch decoder to predict mel-spectrograms and pitch contours from pseudo-phonemes.
• Condition the acoustic model on speaker labels to enable speaker-specific synthesis, with speaker embeddings updated via back-propagation.
• Convert predicted mel-spectrograms and pitch contours into waveforms using HooliGAN, a neural vocoder based on harmonic-plus-noise modeling.
• Train the model end-to-end on the ExVo dataset, using up-sampled pseudo-phoneme sequences aligned with ground-truth mel-spectrograms during training.
• Enable inference with arbitrary speaker labels to transfer timbre while preserving phonetic content.

Experimental results

Research questions

• RQ1 Can HUBERT’s discrete speech units serve as an expressive and meaningful representation for non-speech vocalizations?
• RQ2 Can a multi-speaker text-to-speech system effectively synthesize diverse NSVs with speaker identity control under few-shot conditions?
• RQ3 To what extent do heterogeneous recording conditions distort the learned speaker embedding space in NSV modeling?
• RQ4 Does the model preserve phonetic content when transferring speaker identity across different NSV clips?
• RQ5 What are the key limitations and improvement directions for NSV synthesis using self-supervised speech representations?

Key findings

• HUBERT’s discrete units (treated as pseudo-phonemes) are sufficiently expressive to represent the phonetic content of various NSVs, enabling consistent synthesis across different inputs.
• The model achieves a speaker similarity precision of 0.79 for amusement and 0.51 for awe on the ExVo Generate benchmark, significantly outperforming the baseline (0.49 and 0.46 respectively).
• The Fréchet Inception Distance (FID) for synthesized NSVs is 0.52 ± 0.03 (Syn-1000) and 0.81 ± 0.11 (Syn-100), indicating high perceptual quality and generalization capability.
• Speaker embeddings learned by the model are primarily structured by recording conditions (e.g., cut-off frequency, codec, reverberation) rather than speaker identity, indicating that recording variability dominates the embedding space.
• The model successfully transfers speaker timbre across NSVs while preserving phonetic content, confirming the feasibility of voice conversion from NSV data.
• The model’s performance is limited by recording condition heterogeneity, suggesting that disentangling recording effects from speaker identity could significantly improve robustness.