[论文解读] Multi Modal Information Fusion of Acoustic and Linguistic Data for Decoding Dairy Cow Vocalizations in Animal Welfare Assessment

Bubacarr Jobarteh, Madalina Mincu+2 more|arXiv (Cornell University)|Nov 1, 2024

Food Supply Chain TraceabilityAgricultural and Biological Sciences被引用 3

一句话总结

本研究提出了一种多模态融合框架，结合奶牛接触叫声的声学特征（频率、时长、强度）与语言转录，以评估情绪状态并改善动物福利。通过整合基于自然语言处理（NLP）的转录与声学分析，并采用机器学习模型（随机森林、支持向量机、循环神经网络），该系统成功将叫声分类为高频率的痛苦状态与低频率的满足状态，准确率较高，展示了多源数据融合在精准畜牧养殖中的潜力。

ABSTRACT

Understanding animal vocalizations through multi-source data fusion is crucial for assessing emotional states and enhancing animal welfare in precision livestock farming. This study aims to decode dairy cow contact calls by employing multi-modal data fusion techniques, integrating transcription, semantic analysis, contextual and emotional assessment, and acoustic feature extraction. We utilized the Natural Language Processing model to transcribe audio recordings of cow vocalizations into written form. By fusing multiple acoustic features frequency, duration, and intensity with transcribed textual data, we developed a comprehensive representation of cow vocalizations. Utilizing data fusion within a custom-developed ontology, we categorized vocalizations into high frequency calls associated with distress or arousal, and low frequency calls linked to contentment or calmness. Analyzing the fused multi dimensional data, we identified anxiety related features indicative of emotional distress, including specific frequency measurements and sound spectrum results. Assessing the sentiment and acoustic features of vocalizations from 20 individual cows allowed us to determine differences in calling patterns and emotional states. Employing advanced machine learning algorithms, Random Forest, Support Vector Machine, and Recurrent Neural Networks, we effectively processed and fused multi-source data to classify cow vocalizations. These models were optimized to handle computational demands and data quality challenges inherent in practical farm environments. Our findings demonstrate the effectiveness of multi-source data fusion and intelligent processing techniques in animal welfare monitoring. This study represents a significant advancement in animal welfare assessment, highlighting the role of innovative fusion technologies in understanding and improving the emotional wellbeing of dairy cows.

研究动机与目标

• 开发一种多模态数据融合方法，用于解码奶牛叫声以评估情绪状态。
• 解决在真实农场环境中噪声大、数据多变的背景下准确解读奶牛叫声的挑战。
• 通过智能处理多源数据，提升精准畜牧养殖中的动物福利监测能力。
• 利用声学与语言特征的结合，将奶牛叫声分类为如痛苦或满足等情绪状态。

提出的方法

• 使用自然语言处理（NLP）模型将奶牛叫声转录为文本，以提取语言内容。
• 从音频记录中提取频率、时长和强度等关键声学特征。
• 通过自研本体论融合语言与声学数据，创建叫声的综合表征。
• 应用机器学习模型——随机森林、支持向量机与循环神经网络——基于情绪状态对叫声进行分类。
• 利用情感分析与上下文评估增强融合数据的语言成分。
• 优化模型以应对农场环境中典型的数据质量问题与计算需求。

实验结果

研究问题

• RQ1 声学与语言特征能否有效融合，以提升解码奶牛叫声的准确性？
• RQ2 特定声学特征（如频率与强度）与痛苦或满足等情绪状态之间存在何种关联？
• RQ3 与仅使用声学特征相比，融入转录的语言内容在多大程度上提升了奶牛叫声的分类效果？
• RQ4 在真实农场条件下，哪些机器学习模型在分类奶牛叫声方面表现最佳？
• RQ5 从融合的多模态数据中可否可靠提取出用于支持动物福利评估的情绪指标？

主要发现

• 声学与语言数据的融合显著提升了将奶牛叫声分类为情绪状态的准确性，优于单一模态方法。
• 高频叫声始终与痛苦或警觉状态相关，而低频叫声则与满足或平静状态相关。
• 特定声学特征（如频率升高与独特的声谱模式）被识别为情绪痛苦的可靠指标。
• 随机森林与循环神经网络模型在处理复杂、嘈杂的农场数据方面表现优异，实现了高分类准确率。
• 对转录叫声进行情感分析后，发现20头个体奶牛的情绪模式具有一致性，支持了融合框架的可靠性。
• 自研本体论实现了叫声的结构化表征与分类，提升了系统的可解释性与可扩展性。