[Paper Review] Automatic Sound Event Detection and Classification of Great Ape Calls Using Neural Networks

Zifan Jiang, Adrian Soldati+3 more|arXiv (Cornell University)|Jan 5, 2023

Animal Vocal Communication and BehaviorBiochemistry, Genetics and Molecular Biology9 citations

TL;DR

The paper proposes a neural-network pipeline (wav2vec 2.0 features + LSTM/Transformer) to automatically detect and classify great ape calls from raw continuous audio across chimpanzees, orangutans, and bonobos, achieving high frame-level accuracy and F1-scores. It demonstrates strong generalization of wav2vec 2.0 features without fine-tuning and provides code publicly.

ABSTRACT

We present a novel approach to automatically detect and classify great ape calls from continuous raw audio recordings collected during field research. Our method leverages deep pretrained and sequential neural networks, including wav2vec 2.0 and LSTM, and is validated on three data sets from three different great ape lineages (orangutans, chimpanzees, and bonobos). The recordings were collected by different researchers and include different annotation schemes, which our pipeline preprocesses and trains in a uniform fashion. Our results for call detection and classification attain high accuracy. Our method is aimed to be generalizable to other animal species, and more generally, sound event detection tasks. To foster future research, we make our pipeline and methods publicly available.

Motivation & Objective

• Motivate automatic, accurate detection and classification of great ape calls from raw field recordings.
• Develop a generalizable pipeline that can handle multiple ape lineages with different annotation schemes.
• Evaluate pretrained speech representations (wav2vec 2.0) for cross-species call detection and classification.
• Offer an open-source workflow to facilitate reproducible bioacoustic analysis.

Proposed method

• Convert audio to 16 kHz and segment into 20 ms frames.
• Extract three feature types per frame: raw waveform, spectrogram, and wav2vec 2.0 embeddings.
• Train sequence models (bidirectional LSTM or Transformer encoder) to map frame features to per-frame call labels with optional autoregressive connections.
• Handle class imbalance with weighted loss and augment training with multiple random data splits.
• Evaluate models on three ape datasets (chimpanzee, orangutan, bonobo) with metrics: frame-level accuracy, weighted F1-score, and AUC-PR for binary calls.
• Provide an open-source pipeline for reproducible SED of great ape calls.

Experimental results

Research questions

• RQ1 Can pretrained speech representations (wav2vec 2.0) transfer to biological call detection and classification across great ape species?
• RQ2 What is the performance of LSTM vs Transformer sequence models on frame-level ape call labeling?
• RQ3 Does autoregressive connectivity improve temporal consistency of predictions?
• RQ4 How well does the model generalize across ape species and to binary call-vs-noncall detection?
• RQ5 What are the limits of zero-shot transfer for unseen species (orangutan to bonobo)?

Key findings

• wav2vec 2.0 features with LSTM outperform waveform and spectrogram baselines for chimpanzee data.
• Transformer encoder underperforms vs LSTM on the chimpanzee data given the small data sizes.
• Autoregressive connections improve output consistency and overall performance.
• Extending the model to orangutan and bonobo data yields high frame-level accuracy and F1-scores, with binary-call variant achieving strong performance (AUC-PR up to 0.96).
• Zero-shot transfer from orangutan to bonobo shows potential generalizability (AUC-PR 0.55) indicating room for cross-species SED models.