[Paper Review] Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation

Po-Han Chi, Pei-Hung Chung+5 more|arXiv (Cornell University)|May 18, 2020

Speech Recognition and Synthesis27 references39 citations

TL;DR

Audio ALBERT (AALBERT) is a parameter-efficient self-supervised audio model that shares weights across layers, achieving comparable phoneme and speaker classification performance to Mockingjay with about 91% fewer parameters; intermediate layers encode more phoneme and speaker information.

ABSTRACT

For self-supervised speech processing, it is crucial to use pretrained models as speech representation extractors. In recent works, increasing the size of the model has been utilized in acoustic model training in order to achieve better performance. In this paper, we propose Audio ALBERT, a lite version of the self-supervised speech representation model. We use the representations with two downstream tasks, speaker identification, and phoneme classification. We show that Audio ALBERT is capable of achieving competitive performance with those huge models in the downstream tasks while utilizing 91\% fewer parameters. Moreover, we use some simple probing models to measure how much the information of the speaker and phoneme is encoded in latent representations. In probing experiments, we find that the latent representations encode richer information of both phoneme and speaker than that of the last layer.

Motivation & Objective

• Motivate efficient self-supervised learning for speech representations to enable deployment with limited resources.
• Investigate parameter efficiency by applying ALBERT-style weight sharing to audio transformers.
• Evaluate AALBERT on phoneme and speaker classification to compare with Mockingjay.
• Probe layer-wise representations to understand where phoneme and speaker information is encoded.

Proposed method

• Adapt ALBERT-style weight sharing to a Transformer-based speech model (Audio ALBERT) by sharing parameters across layers.
• Pre-train on masked spectrogram reconstruction to recover log-linear spectrogram using cmvn, with specific masking/noise scheme.
• Compare AALBERT against Mockingjay (12L/6L/3L) in phoneme and speaker classification under feature-extraction and fine-tuning setups.
• Use probing classifiers of varying complexity to assess phoneme and speaker information across layers.
• Analyze attention distributions to understand layer-wise redundancy and information encoding.

(a) Average

Experimental results

Research questions

• RQ1 Can a lite, weight-shared Transformer (AALBERT) match the downstream performance of a larger pre-trained audio model (Mockingjay) while drastically reducing parameters?
• RQ2 Which layers of a pre-trained audio model encode the most phoneme and speaker information?
• RQ3 How does layer depth affect phoneme and speaker probing performance for AALBERT and Mockingjay?
• RQ4 What does attention distribution tell us about parameter sharing and layer redundancy in Audio ALBERT?

Key findings

• AALBERT achieves comparable phoneme classification accuracy to Mockingjay but with 91% fewer parameters.
• AALBERT also matches downstream speaker classification performance while using far fewer parameters than Mockingjay (e.g., 7.4M vs 84.3M for 12L configurations).
• Probing shows intermediate layers contain more phoneme and speaker information than the last layer across both models.
• For both models, intermediate representations outperform the last layer in phoneme and speaker probing tasks.
• Attention analysis reveals redundancy across layers, supporting the rationale for weight sharing to compress parameters without sacrificing representational quality.

(b)