[Paper Review] DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization

Shaoshi Ling, Yuzong Liu|arXiv (Cornell University)|Dec 11, 2020

Speech Recognition and Synthesis34 references58 citations

TL;DR

DeCoAR 2.0 uses Transformer encoders and a vector quantization layer with a diversity objective to learn deep contextualized acoustic representations for semi-supervised speech recognition, achieving competitive WER with limited labeled data compared to baselines.

ABSTRACT

Recent success in speech representation learning enables a new way to leverage unlabeled data to train speech recognition model. In speech representation learning, a large amount of unlabeled data is used in a self-supervised manner to learn a feature representation. Then a smaller amount of labeled data is used to train a downstream ASR system using the new feature representations. Based on our previous work DeCoAR and inspirations from other speech representation learning, we propose DeCoAR 2.0, a Deep Contextualized Acoustic Representation with vector quantization. We introduce several modifications over the DeCoAR: first, we use Transformers in encoding module instead of LSTMs; second, we introduce a vector quantization layer between encoder and reconstruction modules; third, we propose an objective that combines the reconstructive loss with vector quantization diversity loss to train speech representations. Our experiments show consistent improvements over other speech representations in different data-sparse scenarios. Without fine-tuning, a light-weight ASR model trained on 10 hours of LibriSpeech labeled data with DeCoAR 2.0 features outperforms the model trained on the full 960-hour dataset with filterbank features.

Motivation & Objective

• Leverage unlabeled speech data to learn robust acoustic representations for ASR.
• Improve representational quality by replacing LSTMs with Transformers and adding vector quantization.
• Combine reconstruction loss with a diversity objective to train discrete speech representations.
• Demonstrate effectiveness under data-sparse semi-supervised ASR scenarios.
• Analyze the impact of the VQ module on downstream ASR performance.

Proposed method

• Encoder: 1D conv layer followed by Transformer blocks to produce latent z representations (masked frame strategy).
• Quantization: multiple codebooks with Gumbel-Softmax and straight-through estimator map z to a quantized v using discrete codewords.
• Reconstruction: a feed-forward network reconstructs original frames from quantized representations with L1 loss.
• Diversity loss: encourages uniform usage of codebook entries to promote informative linguistic units.
• Joint objective: L = L_recon + alpha * L_div to train the model.
• Semi-supervised downstream: freeze encoder after pretraining; attach to downstream ASR model with no encoder fine-tuning; use CTC loss for ASR.

Experimental results

Research questions

• RQ1 Can Transformer-based encoders with vector quantization produce robust, contextualized acoustic representations from unlabeled data?
• RQ2 Does combining reconstruction loss with a diversity loss improve ASR performance in low-resource labeled-data regimes?
• RQ3 How does DeCoAR 2.0 compare to other representation learning approaches (e.g., wav2vec 2.0, VQ-APC) in semi-supervised LibriSpeech settings?
• RQ4 What is the impact of the VQ layer on downstream ASR accuracy in data-sparse scenarios?

Key findings

• DeCoAR 2.0 with 10 hours of labeled data matches or surpasses systems trained on 960 hours with filterbank features in some conditions.
• In extremely data-sparse scenarios, DeCoAR 2.0 achieves WER of 5.43% (test-clean) and 13.27% (test-other) with 10 hours of labeled data.
• With 1 hour of labeled data, DeCoAR 2.0 achieves WER of 13.75% (test-clean) and 29.13% (test-other).
• Ablation shows the VQ layer benefits ASR performance in the LibriSpeech 10-hour SSL setting (without VQ: 6.29/18.54 vs with VQ: 5.43/13.27).
• DeCoAR 2.0 performs comparably to wav2vec 2.0 across semi-supervised scenarios.