[Paper Review] Masked Autoencoders that Listen

Po-Yao Huang, Xu Hu+6 more|arXiv (Cornell University)|Jul 13, 2022

Music and Audio Processing109 citations

TL;DR

Audio-MAE extends Masked Autoencoders to audio spectrograms, using a high masking ratio and local-decoder attention to learn self-supervised audio representations, achieving state-of-the-art results on six tasks.

ABSTRACT

This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms. Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers. The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram. We find it beneficial to incorporate local window attention in the decoder, as audio spectrograms are highly correlated in local time and frequency bands. We then fine-tune the encoder with a lower masking ratio on target datasets. Empirically, Audio-MAE sets new state-of-the-art performance on six audio and speech classification tasks, outperforming other recent models that use external supervised pre-training. The code and models will be at https://github.com/facebookresearch/AudioMAE.

Motivation & Objective

• Motivate extending masked autoencoder pre-training from images to audio spectrograms to improve self-supervised representations.
• Develop Audio-MAE with a Transformer encoder-decoder architecture that masks spectrogram patches and reconstructs masked content.
• Explore masking strategies (unstructured vs structured) and decoder attention (global vs local) suited to spectrograms.
• Demonstrate that audio-only self-supervised pre-training on AudioSet yields state-of-the-art results across multiple audio and speech tasks.
• Show that pre-training and fine-tuning within the same modality obviates the need for cross-modal transfer learning.

Proposed method

• Encode spectrogram patches with a Transformer encoder after masking a large portion (e.g., 80%) of patches (non-masked patches fed to the encoder).
• Pad encoded patches with learnable mask tokens and decode with a Transformer decoder to reconstruct the input spectrogram.
• In the decoder, apply local attention (shifted windows or a hybrid global+local scheme) to better capture time-frequency locality in audio.
• Use patch-normalized mean squared error as the reconstruction objective; no additional contrastive loss improves performance.
• Fine-tune the encoder for downstream tasks with a lower masking ratio and optional masking to regularize learning, discarding the decoder after pre-training.

Experimental results

Research questions

• RQ1 Can MAE-style masked autoencoding be effectively adapted to audio spectrograms for self-supervised representation learning?
• RQ2 Do local attention mechanisms in the decoder better capture the local time-frequency structure of audio compared to global attention?
• RQ3 How do pre-training masking strategies (unstructured vs structured) and fine-tuning masking strategies affect performance on audio and speech tasks?
• RQ4 Is audio-only pre-training on AudioSet sufficient to achieve state-of-the-art results across multiple audio and speech tasks without cross-modal transfer learning?
• RQ5 What is the impact of model size, patch size, and pre-training data size on Audio-MAE performance?

Key findings

• Audio-MAE achieves state-of-the-art performance on six audio and speech classification tasks with in-domain self-supervised pre-training on AudioSet.
• Pre-training with a high masking ratio (80%) and unstructured masking yields strong performance for pre-training.
• Decoder local attention (shifted window) outperforms vanilla global attention, with hybrid global+local attention also providing gains.
• Fine-tuning with lower masking and structured masking (time+frequency) provides best results for downstream tasks.
• Audible and qualitative reconstructions demonstrate meaningful spectrogram restoration, illustrating the model’s capacity to capture audio structure.