[Paper Review] Low-complexity acoustic scene classification for multi-device audio: analysis of DCASE 2021 Challenge systems

Irene Martín-Morató, Toni Heittola+2 more|arXiv (Cornell University)|May 28, 2021

Music and Audio Processing17 references32 citations

TL;DR

Analyzes DCASE 2021 Task 1A on low-complexity ASC with multiple devices, detailing baseline, submissions, and top systems under a 128 KB model-size constraint.

ABSTRACT

This paper presents the details of Task 1A Acoustic Scene Classification in the DCASE 2021 Challenge. The task targeted development of low-complexity solutions with good generalization properties. The provided baseline system is based on a CNN architecture and post-training quantization of parameters. The system is trained using all the available training data, without any specific technique for handling device mismatch, and obtains an overall accuracy of 47.7%, with a log loss of 1.473. The task received 99 submissions from 30 teams, and most of the submitted systems outperformed the baseline. The most used techniques among the submissions were residual networks and weight quantization, with the top systems reaching over 70% accuracy, and log loss under 0.8. The acoustic scene classification task remained a popular task in the challenge, despite the increasing difficulty of the setup.

Motivation & Objective

• Investigate robustness to device mismatch under strict model-size constraints for acoustic scene classification.
• Evaluate how low-complexity models generalize across seen and unseen devices and cities.
• Identify effective techniques (augmentation, pruning, quantization, distillation) that yield high accuracy under 128 KB.
• Provide insights into architecture choices (residual networks, MobileNet/EfficientNet variants) for constrained devices.

Proposed method

• Use TAU Urban Acoustic Scenes 2020 Mobile dataset with 11 devices (A,B,C,D real; S1-S11 simulated) and 10-class scenes.
• Define 128 KB non-zero parameter limit for model size, excluding feature extraction from complexity calculation.
• Baseline CNN with 40 log mel-band energies, 16-bit quantization post-training, and 10-second clips.
• Evaluate submissions using macro-averaged cross-entropy (log loss) and accuracy with jackknife CI for rankings.
• Analyze top submissions for feature choices, augmentation, and architectures under complexity constraints.
• Compare device seen/unseen performance and city seen/unseen performance to assess generalization.

Experimental results

Research questions

• RQ1 How do low-complexity ASC models perform under multi-device mismatches within a 128 KB parameter budget?
• RQ2 What are the most effective techniques (augmentation, pruning, quantization, distillation) to maximize generalization across devices and cities?
• RQ3 How does performance differ between seen and unseen devices, and between seen and unseen cities?
• RQ4 What architectures best balance accuracy and model size for constrained-device ASC?
• RQ5 To what extent do domain adaptation or learning techniques improve robustness to device mismatch?

Key findings

• Baseline 128 KB-capable CNN with float16 quantization achieves 47.7% accuracy and 1.473 log loss on development data.
• Top submissions reach over 70% accuracy and log loss under 0.8, using residual networks and weight quantization.
• Most top systems stay near the 110–126 KB size range, close to the limit; one tiny model (29 KB) uses aggressive pruning and 16-bit quantization.
• Performance is higher on seen devices (A,B,C,S1–S3) than unseen devices (D,S7–S11), with larger gaps for lower-ranked systems.
• There is a strong correlation between seen and unseen city performance (0.95) but weaker between seen and unseen devices (0.91), indicating device mismatch as the main generalization challenge.
• Residual normalization, domain adversarial training, data augmentation (mixup, specAugment) improve generalization; knowledge distillation is notably effective in ranking.