[Paper Review] All Thresholds Barred: Direct Estimation of Call Density in Bioacoustic Data

Amanda K. Navine, Tom Denton+2 more|arXiv (Cornell University)|Feb 23, 2024

Animal Vocal Communication and Behavior4 citations

TL;DR

This paper proposes a threshold-free method for directly estimating call density in bioacoustic data using Bayesian modeling of classifier scores, avoiding biases from false positives and false negatives. It demonstrates robust performance across varying call densities and model quality, with validation schemes enabling accurate site-level density estimation despite distribution shifts.

ABSTRACT

Passive acoustic monitoring (PAM) studies generate thousands of hours of audio, which may be used to monitor specific animal populations, conduct broad biodiversity surveys, detect threats such as poachers, and more. Machine learning classifiers for species identification are increasingly being used to process the vast amount of audio generated by bioacoustic surveys, expediting analysis and increasing the utility of PAM as a management tool. In common practice, a threshold is applied to classifier output scores, and scores above the threshold are aggregated into a detection count. The choice of threshold produces biased counts of vocalizations, which are subject to false positive/negative rates that may vary across subsets of the dataset. In this work, we advocate for directly estimating call density: The proportion of detection windows containing the target vocalization, regardless of classifier score. Our approach targets a desirable ecological estimator and provides a more rigorous grounding for identifying the core problems caused by distribution shifts -- when the defining characteristics of the data distribution change -- and designing strategies to mitigate them. We propose a validation scheme for estimating call density in a body of data and obtain, through Bayesian reasoning, probability distributions of confidence scores for both the positive and negative classes. We use these distributions to predict site-level densities, which may be subject to distribution shifts. We test our proposed methods on a real-world study of Hawaiian birds and provide simulation results leveraging existing fully annotated datasets, demonstrating robustness to variations in call density and classifier model quality.

Motivation & Objective

• Address the bias introduced by threshold-based detection counts in passive acoustic monitoring (PAM) data.
• Develop a method to directly estimate call density $P(\oplus)$, the proportion of detection windows with target vocalizations, independent of classifier score thresholds.
• Provide a rigorous framework for handling distribution shifts—changes in data distribution across sites or times—that confound traditional threshold-based detection.
• Enable ecologically meaningful inference by estimating site-level call densities that reflect true vocalization rates, not classifier artifacts.
• Support adaptive resource allocation in bioacoustic monitoring by providing confidence intervals and probability distributions for call density estimates.

Proposed method

• Model classifier scores using Bayesian inference to estimate probability distributions for both positive ($P(z|\oplus)$) and negative ($P(z|\ominus)$) classes.
• Use binned validation of classifier outputs to estimate the cumulative distribution of scores across the study population, enabling density estimation without thresholds.
• Apply the law of total probability to relate detection rates $P(z>t)$ to true call density $P(\oplus)$, accounting for false positive and false negative rates.
• Implement a validation scheme with multiple bins (e.g., 4 bins) to distribute classifier scores and reduce bias from extreme logits, especially in noisy environments.
• Use study-level and site-level validation data to estimate site-specific call densities, with substitution assumptions to handle distribution shifts.
• Leverage continuous distribution estimation (e.g., kernel density estimation) as a potential alternative to binned estimation to better capture high-density regions like windy mountain sites.

Figure 1 : Root mean squared error (RMSE) between detection rates at various thresholds and true call density $P(\oplus)$ , using synthetic data and a model with 0.9 ROC-AUC. Notice that the optimal threshold depends on $P(\oplus)$ , which may vary across sites. Dotted lines indicate RMSE for the pr

Experimental results

Research questions

• RQ1 How can call density be estimated directly without relying on arbitrary classifier score thresholds in bioacoustic monitoring?
• RQ2 To what extent does distribution shift—due to environmental, temporal, or behavioral changes—affect threshold-based detection counts and how can this be mitigated?
• RQ3 Can a Bayesian framework for classifier score distributions provide more accurate and robust call density estimates across diverse ecological conditions?
• RQ4 How does the choice of validation binning strategy impact the accuracy of call density estimation, especially in high-noise environments?
• RQ5 Can the proposed method support actionable wildlife management decisions by enabling fine-grained, site-level density estimation with uncertainty quantification?

Key findings

• The proposed threshold-free method achieved robust call density estimation across varying call densities and classifier quality, as validated on real-world Hawaiian bird data.
• Classifier scores from windy mountain sites (e.g., Mauna Kea and Mauna Loa) were heavily concentrated in the top bin, indicating that wind noise acted as a discriminative feature, but this did not compromise the ability to estimate call density.
• With four bins and 200 examples per bin, all ʻUaʻu detections were captured in the top bin, demonstrating high sensitivity despite low overall prevalence.
• The method reduced bias from false positives and false negatives by modeling the full distribution of classifier scores rather than relying on a single threshold.
• Site-level call density estimates were stable and informative, even when distribution shifts occurred due to environmental noise or behavioral variation.
• The approach enabled estimation of call density with low time cost, supporting actionable, real-time wildlife monitoring and evaluation of conservation interventions like mosquito control.

Figure 2 : A schematic of our direct call density estimation method at (A) the study-level using our validation scheme and (B) the site- or covariate-level using computational Strategy 1.