[Paper Review] vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations

Alexei Baevski, Steffen Schneider+1 more|ArXiv.org|Oct 12, 2019

Speech Recognition and Synthesis40 references311 citations

TL;DR

vq-wav2vec learns discrete speech representations via a wav2vec-like self-supervised context-prediction task, quantizes with Gumbel-Softmax or online k-means, and benefits from BERT pre-training on discretized speech for improved ASR performance.

ABSTRACT

We propose vq-wav2vec to learn discrete representations of audio segments through a wav2vec-style self-supervised context prediction task. The algorithm uses either a gumbel softmax or online k-means clustering to quantize the dense representations. Discretization enables the direct application of algorithms from the NLP community which require discrete inputs. Experiments show that BERT pre-training achieves a new state of the art on TIMIT phoneme classification and WSJ speech recognition.

Motivation & Objective

• Motivate learning discrete speech units to enable direct application of NLP techniques to speech data.
• Develop a discretization module for wav2vec that produces fixed-length discrete codes.
• Leverage BERT pre-training on discretized speech to improve downstream ASR performance.

Proposed method

• Extend wav2vec with a vector quantization module to produce discrete codes.
• Use either Gumbel-Softmax or online k-means for codebook-based quantization of dense audio representations.
• Train a Bidirectional Transformer (BERT) on discretized speech to obtain context-rich representations for ASR.
• Evaluate ASR performance with and without language models on WSJ and TIMIT.

Experimental results

Research questions

• RQ1 Can discrete speech representations learned via self-supervised context prediction match or exceed continuous representations for ASR tasks?
• RQ2 Does BERT pre-training on discretized speech improve downstream ASR accuracy on standard benchmarks?
• RQ3 How do different quantization strategies (Gumbel-Softmax vs. k-means) affect performance and codebook utilization?
• RQ4 What are the trade-offs between bitrate, codebook size, and acoustic model performance when discretizing speech?

Key findings

• BERT pre-training on discretized speech achieves state-of-the-art results on WSJ with 2.34 WER on nov92 without language modeling in some setups.
• v q-wav2vec with BERT demonstrates strong results on TIMIT phoneme recognition, achieving 11.64 PER (state of the art at the time).
• Gumbel-Softmax and k-means quantization perform comparatively, with large codebooks narrowing gaps to wav2vec when paired with BERT.
• Discretization enables applying NLP-style sequence models to speech, including standard sequence-to-sequence models, with promising Librispeech results.
• Across experiments, discretized representations often outperform log-mel features and dense wav2vec inputs when used with BERT for ASR.