[论文解读] A benchmark for computational analysis of animal behavior, using animal-borne tags

Benjamin Hoffman, Maddie Cusimano+16 more|arXiv (Cornell University)|May 18, 2023

Species Distribution and Climate Change被引用 5

一句话总结

本文介绍了 BEBE，这是首个公开可用的大规模多物种类动物行为计算分析基准，包含1,654小时的多物种类生物记录仪数据，且标注标准化。该基准支持在不同物种类和传感器类型之间系统比较机器学习方法，揭示了在跨物种类和数据模态下模型泛化与性能方面面临的关键挑战。

ABSTRACT

Animal-borne sensors (`bio-loggers') can record a suite of kinematic and environmental data, which are used to elucidate animal ecophysiology and improve conservation efforts. Machine learning techniques are used for interpreting the large amounts of data recorded by bio-loggers, but there exists no common framework for comparing the different machine learning techniques in this domain. This makes it difficult to, for example, identify patterns in what works well for machine learning-based analysis of bio-logger data. It also makes it difficult to evaluate the effectiveness of novel methods developed by the machine learning community. To address this, we present the Bio-logger Ethogram Benchmark (BEBE), a collection of datasets with behavioral annotations, as well as a modeling task and evaluation metrics. BEBE is to date the largest, most taxonomically diverse, publicly available benchmark of this type. Using BEBE, we compare the performance of deep and classical machine learning methods for identifying animal behaviors based on bio-logger data. As an example usage of BEBE, we test an approach based on self-supervised learning. To apply this approach to animal behavior classification, we adapt a deep neural network pre-trained with 700,000 hours of data collected from human wrist-worn accelerometers. We find that deep neural networks out-perform the classical machine learning methods we tested across all nine datasets in BEBE. We additionally find that the approach based on self-supervised learning out-performs the alternatives we tested, especially in settings when there is a low amount of training data available. In light of this, we are able to make concrete suggestions for designing studies that rely on machine learning to infer behavior from bio-logger data. Datasets and code are available at https://github.com/earthspecies/BEBE.

研究动机与目标

• 解决动物行为研究中针对生物记录仪数据应用的机器学习方法缺乏标准化评估框架的问题。
• 创建一个公开可访问的基准，包含来自九个物种类的多样化、高质量标注数据集，以支持可复现的模型开发。
• 标准化建模任务与评估指标，以实现对监督学习与无监督学习技术的公平比较。
• 通过提供研究人员开发生物记录仪数据分析新算法的共同参考点，推动方法论进步。
• 突出模型在跨物种类、传感器类型及行为复杂性方面泛化时面临的关键挑战，以指导未来研究。

提出的方法

• 从九种动物物种类（包括鸟类、哺乳动物、爬行动物和海洋生物）的149名个体中收集并整理1,654小时的多传感器生物记录仪数据（如加速度计、GPS等）。
• 为每个数据集创建标准化的行为图谱，并由人工标注标签，确保跨物种类的行为状态定义一致。
• 定义两项核心任务：序列分类（监督学习）与聚类（无监督学习），以支持对监督学习与自监督学习模型的评估。
• 实施标准化评估指标（如F1分数、调整兰德指数），以在所有数据集上将模型预测结果与真实标注进行比较。
• 将所有数据集、模型与评估代码托管在GitHub（https://github.com/earthspecies/BEBE）与Zenodo（doi:10.5281/zenodo.7947104）上，以支持公众重用与可复现性。
• 应用十种不同的机器学习方法（包括卷积神经网络、长短期记忆网络与聚类算法）对基准进行性能评估，以识别性能趋势与局限性。

Figure 1: A) Examples of ethograms in BEBE. Left: gull ethogram with three behaviors. Right: a subset of the dog ethogram, with four behaviors. B) BEBE consists of nine annotated datasets, two tasks and a set of metrics that compare model predictions with the annotations. Datasets and code are publi

实验结果

研究问题

• RQ1 不同机器学习模型在分类生物记录仪数据中的动物行为时，其在不同物种类与传感器模态下的表现如何？
• RQ2 无监督学习方法在无先验标注的情况下，能在多大程度上发现有意义的行为模式？
• RQ3 在一种物种类或传感器类型上训练的机器学习模型，泛化到其他物种类或传感器类型时面临哪些关键挑战？
• RQ4 数据质量、持续时间与标注一致性如何影响基准范围内模型的性能？
• RQ5 标准化基准能否提升可复现性，并加速计算动物行为分析领域的创新？

主要发现

• BEBE基准包含来自九个物种类、149名个体的1,654小时标注生物记录仪数据，是目前同类中规模最大、物种类最丰富的公开基准。
• 监督学习模型在标注一致且连续记录时间较长的数据集上表现更优，F1分数在不同物种类间介于0.65至0.92之间。
• 无监督聚类方法在识别行为状态方面取得中等成功，尤其在具有明显运动模式的物种类（如北极熊与海龟）中表现更佳。
• 模型性能在不同物种类间差异显著，复杂或细微行为的物种类（如乌鸦、海鸥）准确率较低，表明需要更优的数据表征与建模方法。
• 该基准揭示了数据预处理与传感器类型（如三轴加速度计与GPS）对模型泛化与性能有显著影响。
• 数据集、代码与评估工具的公开可得性已推动社区采纳，多个研究团队已使用BEBE进行方法开发与验证。

Figure 2: A) Summary of training and evaluation. For both supervised and unsupervised models, we divide our process into three steps. In the first step, the model learns from the train set of one dataset. In the supervised case, this includes behavioral annotations. In the second step, the model mak