[Paper Review] Improving Sound Event Classification by Increasing Shift Invariance in Convolutional Neural Networks

Eduardo Fonseca, Andrés Ferraro+1 more|arXiv (Cornell University)|Jul 1, 2021

Music and Audio Processing22 references4 citations

TL;DR

This paper proposes two novel pooling methods—trainable low-pass filtering (TLPF) and adaptive polyphase sampling (APS)—to enhance shift invariance in CNNs for sound event classification. By inserting these into VGG-based models on the FSD50K dataset, the authors achieve a new state-of-the-art mAP of 0.541 with minimal computational cost and negligible parameter increase.

ABSTRACT

Recent studies have put into question the commonly assumed shift invariance property of convolutional networks, showing that small shifts in the input can affect the output predictions substantially. In this paper, we analyze the benefits of addressing lack of shift invariance in CNN-based sound event classification. Specifically, we evaluate two pooling methods to improve shift invariance in CNNs, based on low-pass filtering and adaptive sampling of incoming feature maps. These methods are implemented via small architectural modifications inserted into the pooling layers of CNNs. We evaluate the effect of these architectural changes on the FSD50K dataset using models of different capacity and in presence of strong regularization. We show that these modifications consistently improve sound event classification in all cases considered. We also demonstrate empirically that the proposed pooling methods increase shift invariance in the network, making it more robust against time/frequency shifts in input spectrograms. This is achieved by adding a negligible amount of trainable parameters, which makes these methods an appealing alternative to conventional pooling layers. The outcome is a new state-of-the-art mAP of 0.541 on the FSD50K classification benchmark.

Motivation & Objective

• Address the lack of shift invariance in CNNs for sound event classification, a problem shown to degrade robustness against small time/frequency shifts in spectrograms.
• Investigate whether architectural modifications to pooling layers can improve robustness and performance without relying on data augmentation or transfer learning.
• Evaluate the effectiveness of anti-aliasing low-pass filtering and adaptive sampling techniques in enhancing shift invariance in CNNs for audio tasks.
• Demonstrate that these modifications yield consistent performance gains across models of varying capacity and under strong regularization.
• Achieve state-of-the-art performance on the FSD50K benchmark using only architectural changes, avoiding reliance on ImageNet pretraining or complex ensembles.

Proposed method

• Introduce a trainable low-pass filter (TLPF) before max-pooling layers to suppress high-frequency components that cause aliasing and reduce shift sensitivity.
• Implement a non-trainable low-pass filter (BlurPool) as a baseline for comparison, using a 5x5 kernel with predefined weights to emulate anti-aliasing.
• Apply adaptive polyphase sampling (APS) to dynamically select the optimal sampling grid based on energy maximization, avoiding fixed subsampling grids that induce shift variance.
• Integrate TLPF and APS into the pooling layers of VGG variants (VGG41 and VGG42), preserving the original network architecture while enhancing invariance.
• Use mixup augmentation as a strong regularizer to ensure generalization and stability across experiments.
• Train models on the FSD50K dataset and evaluate performance using mAP and d′, with ablation studies on shift robustness via controlled spectrogram perturbations.

Experimental results

Research questions

• RQ1 Does lack of shift invariance negatively impact sound event classification performance in CNNs?
• RQ2 Can architectural modifications to pooling layers—specifically low-pass filtering and adaptive sampling—improve shift invariance and classification robustness?
• RQ3 How do TLPF and APS compare in terms of performance gain and computational cost when applied to CNNs for audio tagging?
• RQ4 Can these methods achieve state-of-the-art results without relying on ImageNet pretraining or model ensembling?
• RQ5 To what extent do these modifications enhance robustness against time and frequency shifts in input spectrograms?

Key findings

• The proposed TLPF and APS pooling methods consistently improve sound event classification across models of different capacities on the FSD50K benchmark.
• The combination of TLPF5x5 and APS with l1 norm achieves the highest mAP of 0.541, setting a new state-of-the-art on FSD50K.
• The models with enhanced shift invariance show significantly more stable predictions under time and frequency shifts, as demonstrated by reduced score fluctuations in perturbation experiments.
• The TLPF5x5 method adds only 6k–12k trainable parameters (0.24–0.50% of total), while APS adds no parameters, making both highly efficient.
• The improvements are robust even under strong regularization (mixup), indicating that the gains are not due to overfitting but stem from genuine invariance enhancement.
• The performance of the best model (0.541 mAP) slightly exceeds recent Transformer-based models (0.537 mAP), despite using simpler architecture and no external pretraining.