[Paper Review] Learning Deep Models from Synthetic Data for Extracting Dolphin Whistle Contours

Pu Li, Xiaobai Liua+9 more|arXiv (Cornell University)|May 18, 2020

Marine animal studies overview37 references4 citations

TL;DR

This paper proposes a deep learning method that uses synthetically generated training data to extract dolphin whistle contours from hydrophone recordings. By modeling spectrograms as time-frequency patches and training a neural network to detect archetypical whistle patterns, the approach achieves a 25% improvement in F1-score (0.158 higher) over baseline methods on common and bottlenose dolphin whistles.

ABSTRACT

We present a learning-based method for extracting whistles of toothed whales (Odontoceti) in hydrophone recordings. Our method represents audio signals as time-frequency spectrograms and decomposes each spectrogram into a set of time-frequency patches. A deep neural network learns archetypical patterns (e.g., crossings, frequency modulated sweeps) from the spectrogram patches and predicts time-frequency peaks that are associated with whistles. We also developed a comprehensive method to synthesize training samples from background environments and train the network with minimal human annotation effort. We applied the proposed learn-from-synthesis method to a subset of the public Detection, Classification, Localization, and Density Estimation (DCLDE) 2011 workshop data to extract whistle confidence maps, which we then processed with an existing contour extractor to produce whistle annotations. The F1-score of our best synthesis method was 0.158 greater than our baseline whistle extraction algorithm (~25% improvement) when applied to common dolphin (Delphinus spp.) and bottlenose dolphin (Tursiops truncatus) whistles.

Motivation & Objective

• To address the scarcity of annotated dolphin whistle data in bioacoustics by generating synthetic training samples.
• To develop a deep learning model that accurately detects time-frequency peaks associated with dolphin whistles in noisy underwater recordings.
• To minimize human annotation effort by relying on automated synthesis of training data.
• To improve the performance of whistle contour extraction beyond traditional baseline algorithms.
• To enable scalable, automated analysis of dolphin vocalizations in large-scale bioacoustic datasets.

Proposed method

• The method represents audio signals as time-frequency spectrograms and divides each into localized time-frequency patches.
• A deep neural network is trained to learn archetypical patterns such as frequency-modulated sweeps and crossings from these patches.
• Synthetic training data is generated by combining real background environmental sounds with artificially modeled whistle signals to simulate realistic recording conditions.
• The network predicts time-frequency peaks corresponding to potential whistle contours, which are then processed by an existing contour extractor.
• The training process uses minimal human annotation, relying instead on data synthesis with controlled acoustic parameters.
• The final whistle annotations are derived by post-processing the confidence maps generated by the deep model.

Experimental results

Research questions

• RQ1 Can synthetic data effectively train deep neural networks for dolphin whistle detection with minimal human-annotated data?
• RQ2 How does the performance of a learn-from-synthesis approach compare to traditional whistle extraction baselines?
• RQ3 To what extent can a deep model generalize to detect diverse whistle contours across species like Delphinus and Tursiops truncatus?
• RQ4 What is the impact of synthetic data quality and diversity on model generalization and F1-score in whistle contour extraction?
• RQ5 Can the proposed method achieve high precision and recall in complex, noisy underwater acoustic environments?

Key findings

• The proposed method achieved an F1-score that was 0.158 higher than the baseline algorithm, representing a 25% relative improvement in performance.
• The model demonstrated strong generalization to both common dolphin (Delphinus spp.) and bottlenose dolphin (Tursiops truncatus) whistles.
• The use of synthetic data significantly reduced the need for costly human annotation while maintaining high detection accuracy.
• The confidence maps generated by the model enabled effective contour extraction when processed with an existing contour-finding algorithm.
• The method was validated on a subset of the public DCLDE 2011 workshop dataset, confirming its robustness in real-world conditions.
• The approach was recognized as an invited paper at the International Joint Conference on Neural Networks, indicating its methodological significance.