[Paper Review] Opening the Black Box of wav2vec Feature Encoder

Kwanghee Choi, Eun Jung Yeo|arXiv (Cornell University)|Oct 27, 2022

Speech Recognition and Synthesis4 citations

TL;DR

This paper investigates the internal representations of wav2vec's convolutional feature encoder by feeding it synthesized sine wave signals. It reveals that the latent space encodes fundamental frequency, formants, amplitude, and precise temporal information, forming a metric space where proximity reflects acoustic similarity—offering interpretability into the model's learned representations.

ABSTRACT

Self-supervised models, namely, wav2vec and its variants, have shown promising results in various downstream tasks in the speech domain. However, their inner workings are poorly understood, calling for in-depth analyses on what the model learns. In this paper, we concentrate on the convolutional feature encoder where its latent space is often speculated to represent discrete acoustic units. To analyze the embedding space in a reductive manner, we feed the synthesized audio signals, which is the summation of simple sine waves. Through extensive experiments, we conclude that various information is embedded inside the feature encoder representations: (1) fundamental frequency, (2) formants, and (3) amplitude, packed with (4) sufficient temporal detail. Further, the information incorporated inside the latent representations is analogous to spectrograms but with a fundamental difference: latent representations construct a metric space so that closer representations imply acoustic similarity.

Motivation & Objective

• To understand what acoustic information is encoded within the latent representations of wav2vec's convolutional feature encoder.
• To investigate whether the feature encoder learns discrete acoustic units or continuous representations.
• To analyze the structure of the latent space and determine if it forms a metric space reflecting acoustic similarity.
• To evaluate the contribution of different acoustic components—fundamental frequency, formants, amplitude, and temporal dynamics—to the learned representations.

Proposed method

• Synthesized audio signals composed of simple sine waves were used as inputs to probe the feature encoder.
• The model's latent representations were analyzed across varying sine wave parameters, including frequency, amplitude, and phase.
• The spatial arrangement of embeddings in the latent space was examined to assess whether proximity indicates acoustic similarity.
• Experiments were conducted with controlled combinations of sine waves to isolate the contribution of individual acoustic features.
• The study compared the learned representations to spectrograms to identify similarities and key differences.
• A reductive analysis approach was applied to disentangle the contributions of fundamental frequency, formants, amplitude, and temporal dynamics.

Experimental results

Research questions

• RQ1 What specific acoustic features are encoded in the latent space of the wav2vec feature encoder?
• RQ2 Does the latent space form a metric space where similar acoustic signals produce nearby representations?
• RQ3 How do fundamental frequency, formants, amplitude, and temporal dynamics contribute to the learned representations?
• RQ4 To what extent do the representations resemble spectrograms in structure and information content?
• RQ5 Can the feature encoder be interpreted as learning discrete acoustic units, or is it capturing continuous acoustic properties?

Key findings

• The latent representations encode fundamental frequency as a key component of the learned space.
• Formants are robustly represented in the feature encoder's latent space, indicating phonetic sensitivity.
• Amplitude modulations are preserved and distinguishable in the latent space, reflecting energy dynamics.
• Temporal details are sufficiently resolved to capture fine-grained acoustic variations.
• The latent space forms a metric space where closer embeddings correspond to greater acoustic similarity.
• The representations are analogous to spectrograms but differ fundamentally in their metric structure, enabling direct comparison of acoustic similarity.