[Paper Review] Audio Transformers

Prateek Verma, Jonathan Berger|arXiv (Cornell University)|May 1, 2021

Music and Audio Processing30 references33 citations

TL;DR

This paper presents end-to-end Transformer architectures that operate on raw audio without convolution, achieving state-of-the-art mean average precision on the FSD50K dataset and exploring pooling and multi-scale embeddings to boost performance.

ABSTRACT

Over the past two decades, CNN architectures have produced compelling models of sound perception and cognition, learning hierarchical organizations of features. Analogous to successes in computer vision, audio feature classification can be optimized for a particular task of interest, over a wide variety of datasets and labels. In fact similar architectures designed for image understanding have proven effective for acoustic scene analysis. Here we propose applying Transformer based architectures without convolutional layers to raw audio signals. On a standard dataset of Free Sound 50K,comprising of 200 categories, our model outperforms convolutional models to produce state of the art results. This is significant as unlike in natural language processing and computer vision, we do not perform unsupervised pre-training for outperforming convolutional architectures. On the same training set, with respect mean aver-age precision benchmarks, we show a significant improvement. We further improve the performance of Transformer architectures by using techniques such as pooling inspired from convolutional net-work designed in the past few years. In addition, we also show how multi-rate signal processing ideas inspired from wavelets, can be applied to the Transformer embeddings to improve the results. We also show how our models learns a non-linear non constant band-width filter-bank, which shows an adaptable time frequency front end representation for the task of audio understanding, different from other tasks e.g. pitch estimation.

Motivation & Objective

• Demonstrate that pure Transformer architectures (no convolutions) can outperform CNN baselines on large-scale audio scene understanding.
• Investigate front-end learnable representations that adapt time-frequency analysis to audio tasks.
• Explore pooling and multi-scale embedding strategies to enhance Transformer performance on audio data.
• Show how a learnable front end develops a nonlinear, non-constant bandwidth filter-bank.
• Provide empirical comparisons across architectures on the FSD50K dataset.

Proposed method

• Use a front-end dense layer (2048 then 64) to adapt 1s audio (16 kHz) into Transformer-compatible embeddings.
• Stack Transformer encoders (6 layers, 64-d embeddings, 8 attention heads) with causal multi-head self-attention and feed-forward blocks.
• Incorporate positional encodings (sinusoidal) to retain temporal structure.
• Apply pooling across time after every two Transformer layers to reduce dimensionality and enable hierarchical features.
• Introduce multi-scale embeddings inspired by wavelets via variable windowing and differentiable decompositions.
• Train end-to-end with Adam optimizer and Huber loss on the FSD50K dataset, comparing against CNN baselines.

Experimental results

Research questions

• RQ1 Can a pure Transformer architecture without convolutional layers achieve state-of-the-art performance on large-scale audio understanding tasks?
• RQ2 How do pooling and multi-scale embedding strategies affect Transformer performance on audio data?
• RQ3 What kind of time-frequency representations does a learnable front end develop when trained end-to-end for audio understanding?
• RQ4 How do transformer-based models compare to CNNs across different model scales on FSD50K?
• RQ5 What is the impact of architectural depth and embedding size on audio mAP performance?

Key findings

• Baseline Transformers without convolutions outperform common CNN-based architectures on acoustic scene understanding.
• Large 6-layer Transformer reaches higher mAP than several CNN variants (e.g., 0.525 vs lower CNN scores).
• Pooling (especially average pooling) after every two Transformer layers yields performance gains over the baseline.
• Multi-scale and wavelet-inspired embedding strategies provide further improvements, though exact mAP for the multi-scale variant is not listed.
• The learnable front end discovers a nonlinear, non-constant bandwidth filter bank resembling adaptive time-frequency analysis; filters include sinusoidal, onset detectors, and energy envelopes.
• The front end generalizes across tasks, showing different filter shapes for pitch estimation versus acoustic scene understanding.