[Paper Review] An Empirical Evaluation of Zero Resource Acoustic Unit Discovery

Chunxi Liu, Jinyi Yang+8 more|arXiv (Cornell University)|Feb 5, 2017

Speech Recognition and Synthesis20 references4 citations

TL;DR

This paper proposes an improved acoustic unit discovery (AUD) framework that enhances unsupervised speech representation learning by combining self-supervised linear discriminant analysis (LDA) and multilingual bottleneck (BN) features from a time-delay neural network (TDNN). The method achieves significant gains in downstream tasks like same-different word pair discrimination and document classification, demonstrating that AUD efficacy can be reliably evaluated using alternative zero-resource-compatible data such as word pairs or topic labels when phoneme transcripts are unavailable.

ABSTRACT

Acoustic unit discovery (AUD) is a process of automatically identifying a categorical acoustic unit inventory from speech and producing corresponding acoustic unit tokenizations. AUD provides an important avenue for unsupervised acoustic model training in a zero resource setting where expert-provided linguistic knowledge and transcribed speech are unavailable. Therefore, to further facilitate zero-resource AUD process, in this paper, we demonstrate acoustic feature representations can be significantly improved by (i) performing linear discriminant analysis (LDA) in an unsupervised self-trained fashion, and (ii) leveraging resources of other languages through building a multilingual bottleneck (BN) feature extractor to give effective cross-lingual generalization. Moreover, we perform comprehensive evaluations of AUD efficacy on multiple downstream speech applications, and their correlated performance suggests that AUD evaluations are feasible using different alternative language resources when only a subset of these evaluation resources can be available in typical zero resource applications.

Motivation & Objective

• To improve acoustic unit discovery (AUD) in zero-resource settings where no transcribed speech or linguistic knowledge is available.
• To enhance acoustic feature representations through self-supervised linear discriminant analysis (LDA) using first-pass AUD tokenizations as pseudo-labels.
• To leverage multilingual bottleneck (BN) features from a TDNN to improve cross-lingual generalization in AUD.
• To validate AUD performance using alternative evaluation resources—such as word pairs and topic labels—when orthographic phoneme transcripts are unavailable.
• To demonstrate that intrinsic AUD metrics (like NMI) correlate with extrinsic performance on real speech applications, enabling practical evaluation in zero-resource scenarios.

Proposed method

• Apply self-supervised LDA using AUD-derived tokenizations as pseudo-labels to improve acoustic feature discriminability.
• Train a multilingual time-delay neural network (TDNN) to extract language-independent bottleneck (BN) features for cross-lingual generalization.
• Generate posteriorgram features across discovered HMM units to represent acoustic units for downstream tasks.
• Use DTW scoring with posteriorgrams for the same-different word pair discrimination task, measuring performance via average precision (AP).
• Construct document representations using trigram sequences of discovered acoustic units, applying TF-IDF weighting and L2 normalization for classification.
• Evaluate on document classification (10-fold cross-validation) and clustering (purity and B-Cubed F1) using unsupervised acoustic unit sequences.

Experimental results

Research questions

• RQ1 Can self-supervised LDA improve acoustic unit discovery when no ground-truth labels are available?
• RQ2 Does multilingual bottleneck (BN) feature extraction enhance cross-lingual generalization in zero-resource AUD?
• RQ3 Can downstream speech applications like same-different word pair discrimination and document classification serve as reliable proxies for intrinsic AUD evaluation when phoneme transcripts are unavailable?
• RQ4 To what extent do improvements in extrinsic tasks (e.g., AP, classification accuracy) correlate with intrinsic AUD metrics like NMI?
• RQ5 Can acoustic unit posteriors and trigram representations effectively support discriminative tasks such as document classification and clustering in zero-resource settings?

Key findings

• Combining self-supervised LDA and multilingual BN features yields the best overall AUD performance, demonstrating their complementary nature.
• The same-different task achieves an average precision (AP) of 0.247 on development data using AUD posteriorgrams, significantly outperforming raw MFCCs (AP 0.208).
• Using 600-dimensional HMM-state-level posterior features increases AP to 0.267, indicating higher-resolution representations improve discrimination.
• AUD-based document classification achieves 73% accuracy using LDA- and BN-enhanced features, showing effective representation learning without transcribed data.
• NMI drops only slightly from development (0.571) to evaluation (0.558) data, indicating strong generalization of learned acoustic units.
• Performance on document clustering shows marginal gains on development data, but the trend does not hold on evaluation data, suggesting challenges in unsupervised topic modeling with AUD.