[Paper Review] A Brief Overview of Unsupervised Neural Speech Representation Learning

Lasse Borgholt, Jakob D. Havtorn+3 more|arXiv (Cornell University)|Mar 1, 2022

Speech Recognition and Synthesis126 references5 citations

TL;DR

This paper provides a comprehensive survey of unsupervised neural speech representation learning over the past decade, categorizing methods into self-supervised learning and probabilistic latent variable models. It presents a unified taxonomy, compares model architectures and training objectives, and evaluates representations using downstream tasks like automatic speech recognition and zero-resource benchmarks, highlighting the dominance of masked self-supervised approaches such as wav2vec 2.0.

ABSTRACT

Unsupervised representation learning for speech processing has matured greatly in the last few years. Work in computer vision and natural language processing has paved the way, but speech data offers unique challenges. As a result, methods from other domains rarely translate directly. We review the development of unsupervised representation learning for speech over the last decade. We identify two primary model categories: self-supervised methods and probabilistic latent variable models. We describe the models and develop a comprehensive taxonomy. Finally, we discuss and compare models from the two categories.

Motivation & Objective

• To provide a methodological review of unsupervised neural speech representation learning from 2013 to 2023.
• To classify and compare two dominant categories: self-supervised models and probabilistic latent variable models.
• To develop a comprehensive taxonomy of model architectures and design choices in speech representation learning.
• To analyze evaluation procedures and benchmarks used to assess representation quality, including SUPERB, SLUE, and ZeroSpeech.
• To highlight the evolution and impact of masked self-supervised learning, especially wav2vec 2.0 and its variants, in advancing speech representation performance.

Proposed method

• The paper categorizes models into two primary types: self-supervised learning models and probabilistic latent variable models (LVMs), based on training objectives and architectural design.
• Self-supervised models use pretext tasks such as predicting future frames (e.g., autoregressive modeling) or masked context (e.g., wav2vec 2.0), with a contrastive loss to align representations.
• Latent variable models use variational autoencoders (VAEs) to learn a distribution over latent codes, with reconstruction and KL regularization as key training objectives.
• The authors define a unified notation for input sequences, contextualized representations (c), and latent variables (z), and distinguish between fine-tuned (ftn) and frozen (frz) inference heads.
• A model taxonomy is developed based on architectural components, loss functions, and training objectives, emphasizing structural and functional differences across methods.
• Evaluation is conducted via downstream tasks such as automatic speech recognition, intent classification, speaker verification, and zero-resource challenges like ABX and SLUE.

Experimental results

Research questions

• RQ1 How do self-supervised learning and probabilistic latent variable models differ in architecture, training objective, and representation quality?
• RQ2 What are the key design choices that distinguish successful self-supervised models from latent variable models in speech representation learning?
• RQ3 To what extent do self-supervised models outperform or complement probabilistic latent variable models in downstream speech tasks?
• RQ4 How do evaluation benchmarks like SUPERB, SLUE, and ZeroSpeech assess the semantic, speaker, and phonetic content of learned representations?
• RQ5 What role does masking and contrastive learning play in the success of modern self-supervised speech models like wav2vec 2.0?

Key findings

• Self-supervised learning, particularly masked prediction with contrastive objectives (e.g., wav2vec 2.0), has become the dominant paradigm in unsupervised speech representation learning.
• The SUPERB benchmark demonstrates that representations from self-supervised models achieve state-of-the-art performance on downstream tasks like intent classification, slot filling, and speaker verification.
• The ZeroSpeech challenge shows that self-supervised representations can learn meaningful phonetic and semantic structure without human annotations, especially via the minimal-pair ABX task.
• Latent variable models such as VAEs are effective for representation learning but are less explored in speech than in vision or tabular data, with limited work on how imputation affects learned representations.
• While likelihood and self-supervised loss are poor proxies for representation quality, downstream task performance remains the gold standard for evaluation.
• The success of wav2vec 2.0 and its variants—especially those using masking and contrastive learning—has led to a paradigm shift mirroring masked language modeling in NLP.