[Paper Review] BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacoustics

Lukas Rauch, Raphael Schwinger+9 more|arXiv (Cornell University)|Mar 15, 2024

Animal Vocal Communication and BehaviorBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

BirdSet introduces a standardized, large-scale benchmark for audio classification in avian bioacoustics, unifying diverse open-source datasets into a cohesive framework for multi-class and multi-label classification. It enables reproducible model evaluation through a consistent test set, standardized preprocessing, and integration with Hugging Face, establishing baselines for vision-based and raw-audio transformer models to enhance comparability and accessibility in deep learning for bioacoustics.

ABSTRACT

Deep learning (DL) has greatly advanced audio classification, yet the field is limited by the scarcity of large-scale benchmark datasets that have propelled progress in other domains. While AudioSet is a pivotal step to bridge this gap as a universal-domain dataset, its restricted accessibility and limited range of evaluation use cases challenge its role as the sole resource. Therefore, we introduce BirdSet, a large-scale benchmark dataset for audio classification focusing on avian bioacoustics. BirdSet surpasses AudioSet with over 6,800 recording hours ($\uparrow\!17\%$) from nearly 10,000 classes ($\uparrow\!18 imes$) for training and more than 400 hours ($\uparrow\!7 imes$) across eight strongly labeled evaluation datasets. It serves as a versatile resource for use cases such as multi-label classification, covariate shift or self-supervised learning. We benchmark six well-known DL models in multi-label classification across three distinct training scenarios and outline further evaluation use cases in audio classification. We host our dataset on Hugging Face for easy accessibility and offer an extensive codebase to reproduce our results.

Motivation & Objective

• To address fragmentation and inconsistency in deep learning research for avian bioacoustics by unifying disparate datasets and evaluation protocols.
• To establish a standardized benchmark with consistent test sets and training resources to improve model comparability and reproducibility.
• To support diverse model architectures, including vision-based and raw-audio transformers, through a flexible, community-extensible framework.
• To provide baseline performance metrics and methodological guidelines for future research in passive acoustic monitoring.
• To increase accessibility for newcomers by offering transparent, reusable code and experiment tracking via Weights and Biases.

Proposed method

• The benchmark integrates multiple open-source avian bioacoustics datasets, primarily sourced from Xeno-Canto, into a unified Hugging Face Datasets-compatible format.
• It defines a standardized evaluation pipeline with a fixed set of test datasets and diverse training resources to enable consistent model comparison.
• The framework supports both spectrogram-based models and raw-audio transformers, enabling evaluation across different input modalities.
• A dataset creation tool is provided to facilitate community contributions and ensure extensibility of the benchmark.
• All experiments are tracked using Weights and Biases for transparency, and the codebase is hosted on GitHub for open reuse.
• A comprehensive literature analysis underpins the framework, identifying key challenges in dataset selection, model reliability, training, and evaluation.

Experimental results

Research questions

• RQ1 How do different deep learning architectures perform across standardized multi-class and multi-label bird vocalization classification tasks?
• RQ2 To what extent does model performance vary across diverse datasets with differing recording conditions and labeling schemes?
• RQ3 Can a unified benchmark improve reproducibility and comparability in avian bioacoustics research?
• RQ4 What are the baseline performance metrics for state-of-the-art models on this standardized benchmark?
• RQ5 How does the inclusion of raw-audio models compare to spectrogram-based models in terms of robustness and generalization?

Key findings

• BirdSet establishes a standardized evaluation framework that enables consistent comparison of deep learning models across multiple datasets and tasks.
• Vision-based models and raw-audio transformers achieve competitive performance, with the latter showing promise in handling variable recording conditions.
• The benchmark reveals significant performance variation across datasets, highlighting the need for robust generalization in real-world PAM applications.
• Baseline results are provided for multiple model architectures, offering a reference point for future research and methodological development.
• The integration of diverse datasets through Hugging Face ensures scalability and extensibility for community-driven dataset contributions.
• Transparent experiment tracking via Weights and Biases enhances reproducibility and facilitates methodological refinement.