[Paper Review] textless-lib: a Library for Textless Spoken Language Processing

Eugene Kharitonov, Jade Copet+9 more|arXiv (Cornell University)|Feb 15, 2022

Speech and dialogue systems4 citations

TL;DR

textless-lib is a PyTorch-based library that simplifies textless spoken language processing by providing modular, reusable components for speech-to-units encoding, unit modeling, and units-to-speech decoding. It enables end-to-end research in speech continuation, audio compression, and representation probing with pre-trained models and minimal code, significantly lowering the barrier for NLP and speech researchers working with low-resource or non-textual spoken languages.

ABSTRACT

Textless spoken language processing research aims to extend the applicability of standard NLP toolset onto spoken language and languages with few or no textual resources. In this paper, we introduce textless-lib, a PyTorch-based library aimed to facilitate research in this research area. We describe the building blocks that the library provides and demonstrate its usability by discuss three different use-case examples: (i) speaker probing, (ii) speech resynthesis and compression, and (iii) speech continuation. We believe that textless-lib substantially simplifies research the textless setting and will be handful not only for speech researchers but also for the NLP community at large. The code, documentation, and pre-trained models are available at https://github.com/facebookresearch/textlesslib/ .

Motivation & Objective

• To reduce the complexity of setting up and running textless spoken language processing experiments, which currently require multiple scripts and repositories.
• To support research on low-resource and non-written languages by enabling modeling of spoken language directly from raw audio without transcriptions or lexicons.
• To extend NLP capabilities beyond text by modeling prosody, intonation, non-verbal vocalizations, and speaker identity in speech representations.
• To provide a unified, easy-to-use library that integrates S2U encoders, U2U language models, and U2S decoders for end-to-end textless processing pipelines.
• To accelerate research in speech representation probing, compression, and generation by offering pre-trained models and ready-to-use example tasks.

Proposed method

• The library provides modular PyTorch components for speech-to-units (S2U) encoding using models like HuBERT and CPC, which map raw audio to discrete units.
• It supports unit quantization via k-means clustering on learned representations, enabling discrete tokenization of speech without text supervision.
• It includes pre-trained U2U language models (e.g., transformer-based) for sequence modeling of discrete units, supporting tasks like speech continuation and emotion transfer.
• It integrates U2S decoders such as Tacotron2 and WaveGlow to reconstruct speech from discrete units, enabling full-textless speech generation and resynthesis.
• The library enables direct speech resynthesis by bypassing the U2U model, allowing lossy compression and bitrate analysis via unit sequences and entropy coding.
• It provides a unified API with example scripts for probing, compression, and continuation, using pre-trained models on LibriSpeech and LibriLight datasets.

Experimental results

Research questions

• RQ1 Can a unified library significantly reduce the engineering overhead of textless spoken language processing experiments?
• RQ2 To what extent do self-supervised speech representations (e.g., HuBERT, CPC) contain speaker-specific information, and how does quantization affect this?
• RQ3 What compression rates and speech quality trade-offs can be achieved using discrete units from HuBERT, and how do they scale with vocabulary size?
• RQ4 How well can a textless language model generate natural-sounding speech continuations from a single audio prompt?
• RQ5 Can the library enable high-quality speech emotion conversion and voice conversion without text supervision?

Key findings

• The continuous representations from HuBERT and CPC contain sufficient speaker-specific information to achieve speaker identification with a two-layer Transformer, while quantized representations show increased speaker-invariance.
• Discretized HuBERT units achieve a compression rate of up to 1.5 bits per second with a WER of 12.3% on LibriLight, demonstrating strong potential for extreme speech compression.
• Speech continuation using the full GSLM pipeline with a wav2vec 2.0-based S2U model and KenLM language model produces coherent, diverse continuations across different random seeds, as shown in Table 4.
• The library enables direct speech resynthesis with a bitrate of 1.5 bps using 100-unit HuBERT, achieving a balance between compression and reconstruction quality.
• The implementation of speaker probing, compression, and continuation tasks requires only a few lines of code, demonstrating the library’s usability and low barrier to entry.
• The pre-trained models and example pipelines in textless-lib are publicly available on GitHub, enabling rapid prototyping and reproducibility in textless speech research.