[Paper Review] A multi-device dataset for urban acoustic scene classification

Annamaria Mesaros, Toni Heittola+1 more|arXiv (Cornell University)|Jul 25, 2018

Music and Audio ProcessingComputer Science10 references181 citations

TL;DR

Introduces the TUT Urban Acoustic Scenes 2018 dataset for DCASE 2018, with multi-device recordings across six European cities, and presents a CNN baseline for subtasks on matched and mismatched device conditions.

ABSTRACT

This paper introduces the acoustic scene classification task of DCASE 2018 Challenge and the TUT Urban Acoustic Scenes 2018 dataset provided for the task, and evaluates the performance of a baseline system in the task. As in previous years of the challenge, the task is defined for classification of short audio samples into one of predefined acoustic scene classes, using a supervised, closed-set classification setup. The newly recorded TUT Urban Acoustic Scenes 2018 dataset consists of ten different acoustic scenes and was recorded in six large European cities, therefore it has a higher acoustic variability than the previous datasets used for this task, and in addition to high-quality binaural recordings, it also includes data recorded with mobile devices. We also present the baseline system consisting of a convolutional neural network and its performance in the subtasks using the recommended cross-validation setup.

Motivation & Objective

• Establish a large, variable urban acoustic scene dataset with multi-device recordings to reflect real-world conditions.
• Evaluate baseline CNN performance on high-quality and multi-device recordings.
• Investigate the impact of device mismatch on acoustic scene classification performance.
• Provide development and evaluation pipelines with transparent data splits for fair benchmarking.

Proposed method

• Compute log-mel energies (40 bands) from 10-second segments.
• Train a CNN with two convolutional layers and one fully connected layer using 40x500 input (log-mel energies).
• Use batch normalization and Adam optimizer (lr=0.001) for training.
• Partition data into development/train/test with location-based splits to avoid leakage.
• Provide baselines under three subtasks: A (matched high-quality device), B (mismatched devices with parallel data), and C (external data allowed).
• Time-sync parallel recordings across four devices for robust multi-device evaluation.

Experimental results

Research questions

• RQ1 How does a large, multi-city urban acoustic scene dataset with multi-device recordings affect classification performance?
• RQ2 What is the baseline CNN performance under matched conditions (subtask A) and under device mismatch (subtask B)?
• RQ3 How does device diversity and parallel data impact generalization across devices and evaluation conditions?
• RQ4 Can external data transfer learning (subtask C) improve performance under realistic mismatched conditions?

Key findings

• Baseline CNN achieves 59.7% development accuracy and 61.0% evaluation accuracy for subtask A across 10 scenes.
• For subtask A, per-scene dev/eval performance ranges from 40.4% to 80.5% (dev) and 33.9% to 81.9% (eval), with Public Square being the weakest class.
• Subtask B reveals a substantial drop when testing on devices B, C, and D compared to device A, illustrating device mismatch effects; average performance on B/C is notably lower than on A.
• Average subtask A performance across scenes is 59.7% (dev) and 61.0% (eval).
• Device D (compressed audio) shows particularly poor performance under mismatch, highlighting the impact of recording format on cross-device generalization.