[Paper Review] Acoustic scene classification in DCASE 2020 Challenge: generalization across devices and low complexity solutions

Toni Heittola, Annamaria Mesaros+1 more|arXiv (Cornell University)|May 29, 2020

Music and Audio Processing54 citations

TL;DR

This paper details DCASE 2020 Task 1 with two subtasks: generalization across multiple devices and low-complexity models, reporting dataset, baselines, and top-performing systems.

ABSTRACT

This paper presents the details of Task 1: Acoustic Scene Classification in the DCASE 2020 Challenge. The task consists of two subtasks: classification of data from multiple devices, requiring good generalization properties, and classification using low-complexity solutions. Here we describe the datasets and baseline systems. After the challenge submission deadline, challenge results and analysis of the submissions will be added.

Motivation & Objective

• Motivate robust acoustic scene classification across a wide range of devices in realistic scenarios.
• Present two subtasks: (A) multi-device generalization and (B) low-complexity models with a 500 KB size limit.
• Provide datasets, baselines, and analysis of submissions to guide future research in device-invariant ASC and compact models.

Proposed method

• Introduce TAU Urban Acoustic Scenes 2020 Mobile dataset with real and simulated devices to test generalization.
• Provide a baseline ASC system using Open L3 embeddings and a small CNN/MLP setup per subtask.
• Impose a model size constraint (≤ 500 KB) for Subtask B to encourage low-complexity solutions.
• Evaluate submissions using macro-accuracy and multi-class cross-entropy (log loss).
• Analyze top submissions to identify common strategies such as data augmentation, quantization, pruning, and sparse connectivity.

Experimental results

Research questions

• RQ1 How well do ASC systems generalize across a large set of real and simulated devices?
• RQ2 What strategies enable strong performance under a strict low-complexity (model size) constraint?
• RQ3 What are the main techniques contributing to performance gains in Subtask A and Subtask B?
• RQ4 How do top systems compare to the baseline across seen and unseen devices?

Key findings

• Subtask A achieved up to 76.5% accuracy on the evaluation set, versus 51.4% for the baseline.
• Subtask B achieved up to 96.5% accuracy, versus 89.5% for the baseline, with many systems near the 500 KB limit.
• Data augmentation was the dominant technique to mitigate device mismatch in Subtask A.
• Low-complexity solutions in Subtask B relied on post-training quantization, pruning, sparse connectivity, and low-precision weights (e.g., int8, float16, or 1-bit weights).
• The majority of top Subtask A systems used deep learning architectures (ResNet variants, CNNs) often with ensembles and data augmentation.
• The evaluation dataset for Subtask A included 11 devices across 12 cities; five devices were unseen in training for evaluation.
• The evaluation for Subtask B restricted model size to 500 KB non-zero parameters, promoting compact architectures such as depth-wise separable CNNs and slim models.