[Paper Review] Unsupervised Speech Decomposition via Triple Information Bottleneck

Kaizhi Qian, Shuicheng Yan+3 more|arXiv (Cornell University)|Apr 23, 2020

Speech Recognition and Synthesis39 references43 citations

TL;DR

SpeechSplit is an unsupervised autoencoder that disentangles speech into content, timbre, pitch, and rhythm using three specialized information bottlenecks, enabling blind style transfers across all four components without text labels.

ABSTRACT

Speech information can be roughly decomposed into four components: language content, timbre, pitch, and rhythm. Obtaining disentangled representations of these components is useful in many speech analysis and generation applications. Recently, state-of-the-art voice conversion systems have led to speech representations that can disentangle speaker-dependent and independent information. However, these systems can only disentangle timbre, while information about pitch, rhythm and content is still mixed together. Further disentangling the remaining speech components is an under-determined problem in the absence of explicit annotations for each component, which are difficult and expensive to obtain. In this paper, we propose SpeechSplit, which can blindly decompose speech into its four components by introducing three carefully designed information bottlenecks. SpeechSplit is among the first algorithms that can separately perform style transfer on timbre, pitch and rhythm without text labels. Our code is publicly available at https://github.com/auspicious3000/SpeechSplit.

Motivation & Objective

• Motivate the need to disentangle content, rhythm, pitch, and timbre in speech beyond timbre alone.
• Propose an unsupervised autoencoder framework (SpeechSplit) with three encoders and targeted bottlenecks to separate four speech components.
• Demonstrate that different components can be transferred or modified without relying on text annotations.
• Leverage information bottleneck principles to guide disentanglement and provide insights for general representation learning.

Proposed method

• Introduce three encoders (content, rhythm, pitch) plus a decoder, with an information bottleneck at each encoder output.
• Apply random time-domain resampling to the content and pitch encoders to contaminate rhythm information and create bottlenecks.
• Feed the decoder with all three codes plus speaker identity to reconstruct the spectrogram, enabling blind disentanglement under specified assumptions.
• Use a wavenet vocoder (as in AutoVC) for waveform reconstruction from the decoded spectrogram.
• Train on VCTK without text labels, and evaluate via both subjective MOS and objective pitch metrics (GPE, VDE, FFE).

Experimental results

Research questions

• RQ1 Can speech components content, rhythm, pitch, and timbre be disentangled in an unsupervised setting without text transcriptions?
• RQ2 Do three dedicated bottlenecks force separate encoders to capture distinct components (C, R, F) while timbre information is provided to the decoder separately?
• RQ3 Can SpeechSplit perform independent or combined style transfers (rhythm, pitch, timbre) without parallel data or labels?
• RQ4 What are the subjective and objective performance characteristics of pitch-only versus other conversions compared to AutoVC?

Key findings

• SpeechSplit enables near-separate manipulation of rhythm, pitch, and timbre in spectrograms and pitch contours.
• Pitch-only conversions achieve measurable pitch alignment with target contours (GPE ~1.04%, VDE ~8.14%, FFE ~8.86%).
• Subjective MOS indicates varying difficulty across components, with pitch conversions generally higher than timbre or rhythm when converted alone.
• MOS for different conversion types shows all-three conversion yields lower quality than single-component conversions, and AutoVC remains a timbre-focused baseline.
• Mismatched-content rhythm conversions reveal an anchoring/fill-in-the-blank mechanism that aligns rhythm with content/pitch mappings.
• Removing a component (via zeroing encoder inputs) yields interpretable loss patterns (e.g., blank spectrogram when rhythm is removed).