[Paper Review] Generalization in birdsong classification: impact of transfer learning methods and dataset characteristics

Burooj Ghani, Vincent J. Kalkman+4 more|arXiv (Cornell University)|Sep 21, 2024

Animal Vocal Communication and BehaviorBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This study evaluates transfer learning methods—fine-tuning and knowledge distillation—for bird vocalization classification across diverse datasets and model architectures (CNNs, Transformers). It finds that shallow fine-tuning generalizes better to novel soundscapes, while cross-distillation improves in-domain performance on Xeno-canto data; the authors advocate for comprehensive labeling, including background species and temporal annotations, to enhance model robustness.

ABSTRACT

Animal sounds can be recognised automatically by machine learning, and this has an important role to play in biodiversity monitoring. Yet despite increasingly impressive capabilities, bioacoustic species classifiers still exhibit imbalanced performance across species and habitats, especially in complex soundscapes. In this study, we explore the effectiveness of transfer learning in large-scale bird sound classification across various conditions, including single- and multi-label scenarios, and across different model architectures such as CNNs and Transformers. Our experiments demonstrate that both fine-tuning and knowledge distillation yield strong performance, with cross-distillation proving particularly effective in improving in-domain performance on Xeno-canto data. However, when generalizing to soundscapes, shallow fine-tuning exhibits superior performance compared to knowledge distillation, highlighting its robustness and constrained nature. Our study further investigates how to use multi-species labels, in cases where these are present but incomplete. We advocate for more comprehensive labeling practices within the animal sound community, including annotating background species and providing temporal details, to enhance the training of robust bird sound classifiers. These findings provide insights into the optimal reuse of pretrained models for advancing automatic bioacoustic recognition.

Motivation & Objective

• To evaluate the impact of transfer learning methods—fine-tuning and knowledge distillation—on birdsong classification across diverse datasets and model architectures.
• To assess model generalization performance in complex, real-world soundscapes versus curated training data.
• To investigate the effect of multi-species labeling, particularly incomplete or weak labels for background species, on classifier performance.
• To identify best practices for data curation in bioacoustic datasets to improve model robustness and transferability.
• To determine whether knowledge distillation or fine-tuning yields better performance under varying data and task conditions (single- vs. multi-label)

Proposed method

• Fine-tuned pre-trained audio models (e.g., Wav2Vec 2.0, HuBERT, and Vision Transformers) on Xeno-canto and Dawn Chorus datasets using transfer learning.
• Applied knowledge distillation, including cross-distillation, to transfer knowledge from larger teacher models to smaller student models for improved efficiency and performance.
• Evaluated models under both single-label and multi-label classification settings, with primary labels (foreground species) and secondary labels (background species).
• Used mAP (mean Average Precision) and AUC (Area Under Curve) as primary metrics to assess model performance, with thresholding applied for discrete predictions.
• Analyzed the influence of label completeness and temporal annotation on model confidence and generalization, particularly in low-signal-to-noise environments.
• Compared model performance across different model architectures (CNNs and Transformers) and data distributions to assess robustness and transferability.

Figure 1 : Transfer learning strategies. Light-coloured blocks are neural networks being trained; dark-coloured blocks are ‘frozen’ and unchanging during transfer learning. Shallow fine-tuning (top) uses most of the pretrained model as a fixed feature extractor, retraining the final layer(s) on the

Experimental results

Research questions

• RQ1 How do fine-tuning and knowledge distillation compare in terms of generalization performance on unseen, complex soundscapes versus in-domain data?
• RQ2 What is the impact of including incomplete or weak secondary labels (background species) on model precision, recall, and overall robustness?
• RQ3 Does cross-distillation improve performance on multi-species bird sound classification tasks compared to standard distillation or fine-tuning?
• RQ4 How does the presence of temporal annotations (start/end times of vocalizations) affect model confidence and generalization?
• RQ5 To what extent does label completeness—especially for background species—affect the reliability of automatic bird species detection in real-world monitoring scenarios?

Key findings

• Shallow fine-tuning outperformed knowledge distillation in generalization to novel, complex soundscapes, indicating its robustness and constrained learning behavior.
• Cross-distillation significantly improved in-domain performance on Xeno-canto data, particularly in multi-label scenarios, suggesting its effectiveness for dense labeling tasks.
• The inclusion of secondary (background) species labels increased model recall but reduced precision, indicating a trade-off between sensitivity and accuracy.
• Model confidence was higher and more accurate for primary (foreground) species compared to secondary (background) species, likely due to differences in loudness, distance, and signal-to-noise ratio.
• Despite incomplete labeling, incorporating secondary species labels still yielded positive performance gains, supporting the value of more comprehensive annotation practices.
• mAP remained stable across settings, suggesting that while confidence decreased with multi-label training, the model’s ranking of species likelihoods was preserved.

Figure 2 : Geographic distribution of our data sourced from Xeno-canto.