[Paper Review] Completely Unsupervised Speech Recognition By A Generative Adversarial Network Harmonized With Iteratively Refined Hidden Markov Models

Kuan‐Yu Chen, Che-Ping Tsai+3 more|arXiv (Cornell University)|Apr 8, 2019

Speech Recognition and Synthesis30 references15 citations

TL;DR

This paper proposes a completely unsupervised speech recognition framework that combines a Generative Adversarial Network (GAN) with iteratively refined Hidden Markov Models (HMMs) to learn from unlabeled speech and text data. The GAN's generator produces phoneme sequences by classifying acoustic frames and sampling segment-wise, while the discriminator distinguishes real from generated sequences; HMMs refine segmentation iteratively, yielding a 33.1% phone error rate on TIMIT—8.5% lower than prior state-of-the-art.

ABSTRACT

Producing a large annotated speech corpus for training ASR systems remains difficult for more than 95% of languages all over the world which are low-resourced, but collecting a relatively big unlabeled data set for such languages is more achievable. This is why some initial effort have been reported on completely unsupervised speech recognition learned from unlabeled data only, although with relatively high error rates. In this paper, we develop a Generative Adversarial Network (GAN) to achieve this purpose, in which a Generator and a Discriminator learn from each other iteratively to improve the performance. We further use a set of Hidden Markov Models (HMMs) iteratively refined from the machine generated labels to work in harmony with the GAN. The initial experiments on TIMIT data set achieve an phone error rate of 33.1%, which is 8.5% lower than the previous state-of-the-art.

Motivation & Objective

• To address the challenge of low-resource language ASR by enabling training on purely unlabeled speech and text data.
• To overcome the limitations of prior unsupervised ASR methods that rely on oracle or forced alignment boundaries.
• To develop a joint learning framework where GAN and HMMs iteratively improve each other without any supervision.
• To achieve state-of-the-art performance in completely unsupervised speech recognition using only unlabeled data.

Proposed method

• A GAN architecture is used where the generator includes a frame-wise phoneme classifier and a segment-based sampling process to produce phoneme sequences from acoustic features.
• The discriminator is trained to distinguish real phoneme sequences from text sentences against generated sequences from the generator.
• Initial segmentation boundaries are automatically derived using an unsupervised method (GAS), which are then refined through iterative HMM training on generator outputs.
• The HMMs are retrained in each iteration using the generator’s predicted phoneme sequences, improving segmentation and feedback to the GAN.
• A novel intra-segment loss is introduced to encourage smoothness within phoneme segments, enhancing temporal coherence.
• The framework uses data augmentation and discriminative features (e.g., LDA, MLLT) to improve robustness and performance.

Experimental results

Research questions

• RQ1 Can a GAN-based framework achieve competitive unsupervised speech recognition performance without any labeled data?
• RQ2 How does iterative refinement with HMMs improve the segmentation and phoneme sequence generation in unsupervised ASR?
• RQ3 To what extent does the harmonization between GAN and HMMs reduce error rates compared to standalone GAN or HMM approaches?
• RQ4 Can the proposed method generalize across matched and non-matched text-speech pairs without forced alignment?

Key findings

• The proposed GAN/HMM framework achieved a phone error rate (PER) of 33.1% on the TIMIT dataset in the non-matched case, representing an 8.5% relative reduction over the prior state-of-the-art.
• After three iterations, the method reached a PER of 26.1% in the matched case, outperforming even supervised HMMs trained with 4,000 labeled utterances.
• The ablation study showed that HMMs with triphones, LDA, and MLLT significantly improved performance, while removing data augmentation or the intra-segment loss degraded results.
• Replacing the DNN in the generator with an RNN led to a sharp performance drop (FER: 75.5%, PER: 71.6%), suggesting RNNs may disrupt alignment learning.
• The method converged after three iterations, with consistent improvements across all evaluation metrics in both matched and non-matched settings.
• In comparison to supervised baselines, the unsupervised GAN/HMM approach matched the performance of a standard HMM system when only 15% of the labeled data was available.