[Paper Review] SUPERB: Speech processing Universal PERformance Benchmark

Shu-Wen Yang, Po-Han Chi+18 more|arXiv (Cornell University)|May 3, 2021

Topic ModelingComputer Science40 references51 citations

TL;DR

SUPERB introduces a fixed, frozen-pretrained-model framework to benchmark self-supervised learning representations across 10+ speech tasks with lightweight downstream heads, showing competitive results vs traditional pipelines.

ABSTRACT

Self-supervised learning (SSL) has proven vital for advancing research in natural language processing (NLP) and computer vision (CV). The paradigm pretrains a shared model on large volumes of unlabeled data and achieves state-of-the-art (SOTA) for various tasks with minimal adaptation. However, the speech processing community lacks a similar setup to systematically explore the paradigm. To bridge this gap, we introduce Speech processing Universal PERformance Benchmark (SUPERB). SUPERB is a leaderboard to benchmark the performance of a shared model across a wide range of speech processing tasks with minimal architecture changes and labeled data. Among multiple usages of the shared model, we especially focus on extracting the representation learned from SSL due to its preferable re-usability. We present a simple framework to solve SUPERB tasks by learning task-specialized lightweight prediction heads on top of the frozen shared model. Our results demonstrate that the framework is promising as SSL representations show competitive generalizability and accessibility across SUPERB tasks. We release SUPERB as a challenge with a leaderboard and a benchmark toolkit to fuel the research in representation learning and general speech processing.

Motivation & Objective

• Provide a standard, comprehensive benchmark to assess generalization and reusability of SSL speech representations across diverse tasks.
• Evaluate the effectiveness of a frozen, shared pretrained model with lightweight, task-specific heads.
• Compare SSL representations against traditional features (FBANK) across a broad task set.
• Promote open, reproducible evaluation with an open-source toolkit and leaderboard.

Proposed method

• Partition tasks into content, speaker, semantics, and paralinguistics with ten tasks and standardized datasets.
• Use a frozen, shared SSL pretrained model as universal representation encoder.
• Attach lightweight, task-specific prediction heads and train only these heads (plus small downstream components).
• Extract multi-layer representations and combine via weighted sums for downstream inputs.
• Evaluate a wide range of SSL models spanning generative, discriminative, and multi-task pretraining.
• Provide a benchmark toolkit and online leaderboard for submissions and reproducibility.

Experimental results

Research questions

• RQ1 Can a single pretrained SSL model serve as a universal representation encoder across diverse speech tasks with minimal downstream training?
• RQ2 How do different SSL pretraining paradigms (generative, discriminative, multi-task) compare when used in a frozen-representation, lightweight-head setup?
• RQ3 Do SSL representations outperform traditional FBANK features across these tasks under constrained downstream tuning?
• RQ4 What is the relative performance and generalization of SSL models on content, speaker, semantics, and paralinguistics tasks?

Key findings

• SSL representations (e.g., wav2vec 2.0 and HuBERT) achieve competitive results with lightweight task heads across multiple SUPERB tasks.
• FBANK features require more downstream complexity to compete; SSL representations often outperform FBANK with linear or simple downstream models.
• Certain SSL models excel in specific tasks (PR, IC, QbE, SF) and demonstrate notable improvements in end-to-end ASR and QbE contexts.
• The framework demonstrates strong generalization and reusability of SSL representations for speech processing tasks.
• HuBERT and wav2vec 2.0 provide particularly strong performance across several tasks, including QbE and SF.
• The approach enables easier ASR system development by leveraging robust SSL representations with minimal downstream customization.