[Paper Review] The Zero Resource Speech Challenge 2020: Discovering discrete subword and word units

Ewan Dunbar, Julien Karadayi+7 more|arXiv (Cornell University)|Oct 12, 2020

Speech Recognition and SynthesisComputer Science28 references19 citations

TL;DR

This paper presents the Zero Resource Speech Challenge 2020, which advances unsupervised speech representation learning by discovering discrete subword and word units from raw audio without any textual supervision. It achieves significant improvements in speech synthesis quality and spoken term discovery, with top systems approaching phonemic-level unit quality at low bitrates, demonstrating progress toward human-like speech learning without labels.

ABSTRACT

We present the Zero Resource Speech Challenge 2020, which aims at learning speech representations from raw audio signals without any labels. It combines the data sets and metrics from two previous benchmarks (2017 and 2019) and features two tasks which tap into two levels of speech representation. The first task is to discover low bit-rate subword representations that optimize the quality of speech synthesis; the second one is to discover word-like units from unsegmented raw speech. We present the results of the twenty submitted models and discuss the implications of the main findings for unsupervised speech learning.

Motivation & Objective

• To develop unsupervised speech representation learning methods that discover discrete subword and word units from raw audio without any textual resources.
• To evaluate the quality of discovered units in terms of speech synthesis performance, unit bitrate, and linguistic accuracy using gold phoneme and word transcriptions.
• To advance the state of the art in zero-resource speech technology by combining and reopening prior benchmarks focused on acoustic unit discovery and spoken term discovery.
• To explore whether high-quality, low-bitrate discrete units—akin to phonemes or words—can be learned purely from raw audio, mimicking human language acquisition.
• To assess the trade-offs between unit quality, bitrate, coverage, and segmentation accuracy in unsupervised settings.

Proposed method

• Participants trained models on unlabelled raw audio from English and Indonesian, using three data sets: Train Voice (for voice cloning), Train Unit Discovery (for speaker-independent units), and Test (for evaluation).
• For speech synthesis, models encoded raw audio into discrete units (pseudo-text), which were then decoded into waveforms using a vocoder; synthesis quality was evaluated via human assessment of CER, MOS, and voice similarity.
• Unit quality was measured using ABX error rate, and bitrate was computed from the entropy of the distinct unit vectors in the test set.
• For spoken term discovery, models detected word-like units in continuous speech using clustering and dynamic time warping, with performance evaluated via NED (normalized edit distance) and F-score for boundary detection.
• The challenge used standardized metrics and baselines: a LSH-based pair-matching baseline and a supervised topline using unigram grammar and textual transcriptions.
• All evaluations were conducted using gold phonetic and word transcriptions, with strict quality control via catch trials and human evaluation.

Experimental results

Research questions

• RQ1 Can discrete subword units be discovered from raw audio alone that enable high-quality, low-bitrate speech synthesis without any textual supervision?
• RQ2 To what extent can unsupervised models discover word-like units in unsegmented speech, and how do they compare to supervised baselines in terms of coverage and accuracy?
• RQ3 What is the trade-off between unit quality, bitrate, and synthesis naturalness in unsupervised speech representation learning?
• RQ4 Can clustering-based approaches for spoken term discovery achieve better balance between matching quality and coverage than previous methods?
• RQ5 How do end-to-end models based on discrete autoencoders compare to traditional pipeline approaches in unit discovery and synthesis?

Key findings

• Top systems achieved a 20% reduction in ABX error rate compared to the previous year’s best models, indicating improved unit quality at comparable bitrates.
• The best systems produced speech synthesis with a mean opinion score (MOS) of 4.0 or higher and a character error rate (CER) below 15%, demonstrating high comprehensibility and naturalness.
• Systems using self-expressing autoencoders achieved boundary F-scores comparable to the previous state of the art in spoken term discovery, indicating strong segmentation performance.
• The R1 and R2 systems using probabilistic dynamic time warping improved the trade-off between coverage and matching quality, with R1 achieving high coverage and R2 achieving low NED (normalized edit distance).
• Despite progress, the best unsupervised discrete codes remained an order of magnitude more detailed than phonemic representations, indicating a gap to full phonemic-level abstraction.
• The supervised topline system demonstrated that a low-bitrate, high-quality discrete code is achievable with textual supervision, highlighting the challenge of replicating this in an unsupervised setting.