[Paper Review] PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition

Qiuqiang Kong, Yin Cao+4 more|arXiv (Cornell University)|Dec 21, 2019

Music and Audio ProcessingComputer Science48 references73 citations

TL;DR

The paper trains large-scale pretrained audio neural networks (PANNs) on AudioSet and demonstrates state-of-the-art performance for audio tagging, with successful transfer to multiple audio pattern recognition tasks using various architectures and input representations.

ABSTRACT

Audio pattern recognition is an important research topic in the machine learning area, and includes several tasks such as audio tagging, acoustic scene classification, music classification, speech emotion classification and sound event detection. Recently, neural networks have been applied to tackle audio pattern recognition problems. However, previous systems are built on specific datasets with limited durations. Recently, in computer vision and natural language processing, systems pretrained on large-scale datasets have generalized well to several tasks. However, there is limited research on pretraining systems on large-scale datasets for audio pattern recognition. In this paper, we propose pretrained audio neural networks (PANNs) trained on the large-scale AudioSet dataset. These PANNs are transferred to other audio related tasks. We investigate the performance and computational complexity of PANNs modeled by a variety of convolutional neural networks. We propose an architecture called Wavegram-Logmel-CNN using both log-mel spectrogram and waveform as input feature. Our best PANN system achieves a state-of-the-art mean average precision (mAP) of 0.439 on AudioSet tagging, outperforming the best previous system of 0.392. We transfer PANNs to six audio pattern recognition tasks, and demonstrate state-of-the-art performance in several of those tasks. We have released the source code and pretrained models of PANNs: https://github.com/qiuqiangkong/audioset_tagging_cnn.

Motivation & Objective

• Introduce PANNs trained on AudioSet (1.9M clips, 527 classes) for broad audio tagging and pattern recognition.
• Evaluate a wide range of CNN architectures (CNN, ResNet, MobileNet, One-dimensional CNNs) for AudioSet tagging.
• Propose Wavegram-Logmel-CNN that combines time-domain waveforms and log-mel features for improved tagging.
• Describe data processing techniques (balancing, Mixup, SpecAugment) to improve generalization.
• Demonstrate transfer of PANNs to diverse audio tasks beyond tagging and release code/models.

Proposed method

• Train PANNs on raw AudioSet audio with binary cross-entropy loss for multi-label tagging.
• Explore architectures including CNN14, VGG-like CNNs (6/10/14 layers), ResNets (ResNet22/38/54), MobileNets (V1/V2), and one-dimensional CNNs (DaiNet, LeeNet, Res1dNet).
• Introduce Wavegram-CNN: learn a time-frequency Wavegram from waveform using 1D CNNs, then apply 2D CNNs (CNN14) on the Wavegram.
• Propose Wavegram-Logmel-CNN by fusing Wavegram and log-mel spectrogram inputs along the channel dimension.
• Data processing: balanced sampling in mini-batches, mixup augmentation on log-mel or waveform, and SpecAugment on log-mel spectrograms.
• Transfer strategies: train from scratch, use PANN as feature extractor (frozen), or fine-tune the PANN on new tasks.

Experimental results

Research questions

• RQ1 How well do PANN architectures trained on AudioSet perform on AudioSet tagging compared to prior state-of-the-art systems?
• RQ2 How do input representations (log-mel spectrogram vs waveform vs Wavegram) and network architectures affect tagging performance and efficiency?
• RQ3 Can PANNs pretrained on AudioSet transfer effectively to other audio pattern recognition tasks (e.g., acoustic scene classification, music tagging, speech emotion) and with what strategies?

Key findings

• CNN14-based PANNs achieved an mAP of 0.431 on AudioSet tagging, outperforming previous state-of-the-art of 0.392.
• Wavegram-Logmel-CNN further improves performance over CNN14 and MobileNetV1 baselines.
• Data balancing and augmentation (Mixup, SpecAugment) substantially improve mAP (e.g., 0.431 with bal+mixup on 1.9M samples).
• CNN-based PANNs trained on AudioSet transfer effectively to other tasks, achieving strong results relative to task-specific baselines.
• The authors release source code and pretrained PANN models for reuse.
• The best overall configuration (CNN14 backbone with Wavegram-Logmel input) yields superior performance across multiple metrics and tasks compared to embedding-feature-based approaches.