Skip to main content
QUICK REVIEW

[Paper Review] AST: Audio Spectrogram Transformer

Yuan Gong, Yu-An Chung|arXiv (Cornell University)|Apr 5, 2021
Music and Audio ProcessingComputer Science35 references31 citations
TL;DR

The paper introduces AST, a convolution-free, purely attention-based model for audio classification, and shows it achieves state-of-the-art results on AudioSet, ESC-50, and Speech Commands V2, aided by transferring pretrained Vision Transformer weights.

ABSTRACT

In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels. To better capture long-range global context, a recent trend is to add a self-attention mechanism on top of the CNN, forming a CNN-attention hybrid model. However, it is unclear whether the reliance on a CNN is necessary, and if neural networks purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST), the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.

Motivation & Objective

  • Evaluate whether a CNN is necessary for strong audio classification performance.
  • Develop a purely attention-based model that operates on spectrograms and captures long-range context.
  • Explore transfer learning from Vision Transformer (ViT) pretrained on ImageNet to AST.
  • Compare AST with CNN-based and CNN-attention hybrid models across multiple audio datasets.

Proposed method

  • Split log-Mel spectrograms into overlapping 16x16 patches and linearly project to 768-d patch embeddings.
  • Add trainable 768-d positional embeddings and prepend a [CLS] token to form input to a 12-layer, 12-head Transformer encoder.
  • Use the [CLS] token output for classification via a linear layer with sigmoid activation.
  • Adapt ImageNet-pretrained ViT weights to AST by averaging input-channel weights and interpolating positional embeddings for variable input lengths.
  • Train with ImageNet pretraining, data augmentation (mixup, SpecAugment-like masking), and model averaging/ensembles to improve performance.
  • Evaluate transfer learning effects across balanced/full AudioSet and on ESC-50 and Speech Commands V2.
Figure 1: The proposed audio spectrogram transformer (AST) architecture. The 2D audio spectrogram is split into a sequence of 16 $\times$ 16 patches with overlap, and then linearly projected to a sequence of 1-D patch embeddings. Each patch embedding is added with a learnable positional embedding. A
Figure 1: The proposed audio spectrogram transformer (AST) architecture. The 2D audio spectrogram is split into a sequence of 16 $\times$ 16 patches with overlap, and then linearly projected to a sequence of 1-D patch embeddings. Each patch embedding is added with a learnable positional embedding. A

Experimental results

Research questions

  • RQ1Can a convolution-free, purely attention-based model match or surpass CNN-based and CNN-attention hybrid architectures for audio classification?
  • RQ2Does transfer learning from Vision Transformer pretrained on ImageNet improve audio classification performance when ported to AST?
  • RQ3How does AST perform across diverse audio datasets (AudioSet, ESC-50, Speech Commands V2) with variable input lengths?
  • RQ4What design choices (patch size/overlap, positional embedding adaptation, patch shape) most affect AST performance?

Key findings

  • AST achieves state-of-the-art results on AudioSet with ensemble full-set mAP of 0.485 and single-model mAP of 0.459.
  • AST outperforms prior CNN and CNN-attention hybrids on AudioSet in both balanced and full settings.
  • For ESC-50, AST-S (ImageNet pretraining only) attains 88.7% accuracy and AST-P reaches 95.6% accuracy, surpassing SOTA under both settings.
  • For Speech Commands V2, AST-S achieves 98.11% and AST-P 97.88% accuracy, with ImageNet+AudioSet pretraining not always necessary for best performance on this task.
  • ImageNet pretraining markedly boosts performance, especially with smaller in-domain data; DeiT-based weights yielded the best AudioSet results among ViT weights evaluated.
  • Positional embedding adaptation via cut-and-bilinear interpolation is important for leveraging ViT priors in AST.

Read this paper

Open the full text in the Nubint reader and ask about any passage as you read.

Read this paper on Nubint

This review was generated by AI from the original paper. Please check the paper before citing it.