[Paper Review] Rethinking CNN Models for Audio Classification

Kamalesh Palanisamy, Dipika Singhania+1 more|arXiv (Cornell University)|Jul 22, 2020

Music and Audio ProcessingComputer Science62 references108 citations

TL;DR

The paper shows ImageNet-pretrained CNNs (DenseNet, ResNet, Inception) fine-tuned on Mel-spectrograms achieve state-of-the-art on ESC-50 and UrbanSound8K, and competitive results on GTZAN, with ensembles improving robustness.

ABSTRACT

In this paper, we show that ImageNet-Pretrained standard deep CNN models can be used as strong baseline networks for audio classification. Even though there is a significant difference between audio Spectrogram and standard ImageNet image samples, transfer learning assumptions still hold firmly. To understand what enables the ImageNet pretrained models to learn useful audio representations, we systematically study how much of pretrained weights is useful for learning spectrograms. We show (1) that for a given standard model using pretrained weights is better than using randomly initialized weights (2) qualitative results of what the CNNs learn from the spectrograms by visualizing the gradients. Besides, we show that even though we use the pretrained model weights for initialization, there is variance in performance in various output runs of the same model. This variance in performance is due to the random initialization of linear classification layer and random mini-batch orderings in multiple runs. This brings significant diversity to build stronger ensemble models with an overall improvement in accuracy. An ensemble of ImageNet pretrained DenseNet achieves 92.89% validation accuracy on the ESC-50 dataset and 87.42% validation accuracy on the UrbanSound8K dataset which is the current state-of-the-art on both of these datasets.

Motivation & Objective

• Demonstrate that ImageNet-pretrained CNNs can serve as strong baselines for audio classification using mel-spectrogram inputs.
• Quantify the benefit of pretrained weights versus random initialization across multiple datasets.
• Analyze how pretrained weights change during fine-tuning and identify which network parts matter most for audio tasks.
• Provide qualitative insight into what CNNs learn from spectrograms via gradient-based visualizations.
• Show that deep ensembles yield higher accuracy and robustness across datasets.

Proposed method

• Use ImageNet-pretrained DenseNet-201, ResNet, and Inception models fine-tuned on mel-spectrogram inputs derived from audio datasets.
• Convert mel-spectrograms to a three-channel input (either replicated single spectrogram or multi-window-channel approach) and apply standard augmentations (time-stretch, pitch-shift).
• Train models on ESC-50, UrbanSound8K, and GTZAN with tuned hyperparameters (Adam, lr=1e-4, weight decay 1e-3).
• Evaluate single models and ensembles (M=5) by averaging softmax outputs to assess performance gains from ensembling.
• Perform transfer-learning analyses: weights change, partial weight fusion/freezing, and model cutoff to identify where pretrained knowledge helps most.

Experimental results

Research questions

• RQ1 Do ImageNet-pretrained CNNs fine-tuned on mel-spectrograms outperform training from scratch on common audio classification datasets?
• RQ2 Which parts of the pretrained networks retain useful audio representations after fine-tuning, and how does freezing or partially transferring weights affect performance?
• RQ3 Can simple mel-spectrogram inputs with standard CNN backbones achieve state-of-the-art results on ESC-50 and UrbanSound8K using transfer learning from ImageNet?
• RQ4 Does ensembling multiple fine-tuned pretrained models yield robust gains across datasets?
• RQ5 What do gradient-based visualizations reveal about how CNNs interpret spectrogram inputs?

Key findings

• Pretrained weights consistently improve performance over random initialization across ESC-50, UrbanSound8K, and GTZAN (e.g., ~20% gain on ESC-50, ~10% on UrbanSound8K, and >3% on GTZAN).
• An ensemble of ImageNet-pretrained DenseNet achieves 92.89% on ESC-50 and 87.42% on UrbanSound8K (state-of-the-art at the time).
• Block3 (middle stage) of the networks is crucial for transferring knowledge from ImageNet to audio; freezing or removing this region substantially degrades performance.
• Integrated Gradients visualizations show the model focuses on high-energy regions in spectrograms, indicating learned edge-like boundaries around sound events.
• Weight-change analyses (SVCCA) indicate initial layers retain much of their pretrained features after fine-tuning, while middle layers undergo more task-specific adaptation.
• Ensembling five independently trained models yields roughly +2% absolute gains on ESC-50 and UrbanSound8K (slight variation on GTZAN).