[Paper Review] Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data

Wei-Ning Hsu, Yu Zhang+1 more|arXiv (Cornell University)|Sep 22, 2017

Speech Recognition and Synthesis26 references150 citations

TL;DR

Proposes a factorized hierarchical variational autoencoder (FHVAE) for unsupervised disentanglement of sequence-level and segment-level attributes in sequential data, demonstrated on speech with improvements over i-vector baselines and ASR performance gains in mismatched conditions.

ABSTRACT

We present a factorized hierarchical variational autoencoder, which learns disentangled and interpretable representations from sequential data without supervision. Specifically, we exploit the multi-scale nature of information in sequential data by formulating it explicitly within a factorized hierarchical graphical model that imposes sequence-dependent priors and sequence-independent priors to different sets of latent variables. The model is evaluated on two speech corpora to demonstrate, qualitatively, its ability to transform speakers or linguistic content by manipulating different sets of latent variables; and quantitatively, its ability to outperform an i-vector baseline for speaker verification and reduce the word error rate by as much as 35% in mismatched train/test scenarios for automatic speech recognition tasks.

Motivation & Objective

• Leverage multi-scale information in sequential data to learn disentangled, interpretable latent factors without supervision.
• Factorize attributes into sequence-level (z2) and segment-level (z1) latent variables using sequence-dependent and sequence-independent priors.
• Enable scalable inference at segment level to handle long sequences while preserving temporal structure.
• Demonstrate learning of disentangled representations through qualitative analyses and quantitative ASR and speaker verification tasks.

Proposed method

• Introduce a Factorized Hierarchical Variational Autoencoder (FHVAE) with two latent variable sets: z1 (segment-level) and z2 (sequence-level) plus an s-vector μ2 for each sequence.
• P(z1) is a sequence-independent prior; P(z2|μ2) is a sequence-dependent prior; P(x|z1,z2) is Gaussian with mean/variance given by neural nets fμx(·,·) and fσ2x(·,·).
• Encoders qφ(z1|x,z2), qφ(z2|x), and qφ(μ2) are diagonal Gaussians parameterized by LSTM/MLP networks; a discriminative objective α log p(i|z2) encourages z2 to encode sequence-level attributes.
• Utilize a Seq2Seq-FHVAE architecture to enable segment-level optimization and scalability by evaluating the segment-level lower bound rather than full sequences.
• Provide testing-time μ2 inference via a closed-form approximation (Eq. 5) using z2 posteriors, enabling usage as an utterance-level representation.

Experimental results

Research questions

• RQ1 Can a factorized hierarchical VAE learn disentangled, interpretable sequence-level and segment-level latent factors from sequential data without supervision?
• RQ2 Do the segment-level (z1) and sequence-level (z2) latents align with linguistic content and speaker/ channel attributes respectively in speech?
• RQ3 Do the learned latent variables improve speaker verification and domain-invariant ASR compared to baselines such as i-vectors and β-VAE features?
• RQ4 Is segment-level latent representation robust to domain mismatches in ASR and useful for voice conversion or denoising tasks?

Key findings

• On speaker verification (TIMIT), μ2 (16–32 dim) outperforms i-vector baselines in Raw and LDA settings, achieving as low as 2.38% EER with 32-dim μ2 and α=10^1.
• In domain-mismatch ASR (Aurora-4), latent z1 features reduce WER significantly across noisy/channel conditions and outperform FBank and β-VAE baselines in several mismatched domains.
• Replacing z2 with a different speaker can yield voice-converted outputs preserving linguistic content, evidencing disentanglement of speaker and content.
• Qualitative analyses show z1 capturing segment-level linguistic content and z2 capturing sequence-level attributes, enabling denoising by swapping μ2-related representations.