[Paper Review] Multi Modal Information Fusion of Acoustic and Linguistic Data for Decoding Dairy Cow Vocalizations in Animal Welfare Assessment

Bubacarr Jobarteh, Madalina Mincu+2 more|arXiv (Cornell University)|Nov 1, 2024

Food Supply Chain TraceabilityAgricultural and Biological Sciences3 citations

TL;DR

This study proposes a multi-modal fusion framework combining acoustic features (frequency, duration, intensity) and linguistic transcriptions of dairy cow contact calls to assess emotional states and improve animal welfare. By integrating NLP-based transcription with acoustic analysis and employing machine learning models (Random Forest, SVM, RNN), the system successfully classified vocalizations into distress (high-frequency) and contentment (low-frequency) states with high accuracy, demonstrating the potential of multi-source data fusion in precision livestock farming.

ABSTRACT

Understanding animal vocalizations through multi-source data fusion is crucial for assessing emotional states and enhancing animal welfare in precision livestock farming. This study aims to decode dairy cow contact calls by employing multi-modal data fusion techniques, integrating transcription, semantic analysis, contextual and emotional assessment, and acoustic feature extraction. We utilized the Natural Language Processing model to transcribe audio recordings of cow vocalizations into written form. By fusing multiple acoustic features frequency, duration, and intensity with transcribed textual data, we developed a comprehensive representation of cow vocalizations. Utilizing data fusion within a custom-developed ontology, we categorized vocalizations into high frequency calls associated with distress or arousal, and low frequency calls linked to contentment or calmness. Analyzing the fused multi dimensional data, we identified anxiety related features indicative of emotional distress, including specific frequency measurements and sound spectrum results. Assessing the sentiment and acoustic features of vocalizations from 20 individual cows allowed us to determine differences in calling patterns and emotional states. Employing advanced machine learning algorithms, Random Forest, Support Vector Machine, and Recurrent Neural Networks, we effectively processed and fused multi-source data to classify cow vocalizations. These models were optimized to handle computational demands and data quality challenges inherent in practical farm environments. Our findings demonstrate the effectiveness of multi-source data fusion and intelligent processing techniques in animal welfare monitoring. This study represents a significant advancement in animal welfare assessment, highlighting the role of innovative fusion technologies in understanding and improving the emotional wellbeing of dairy cows.

Motivation & Objective

• To develop a multi-modal data fusion approach for decoding dairy cow vocalizations to assess emotional states.
• To address the challenge of accurately interpreting cow vocalizations in real-world farm environments with noisy and variable data.
• To enhance animal welfare monitoring in precision livestock farming through intelligent processing of multi-source data.
• To classify cow vocalizations into emotional states such as distress or contentment using combined acoustic and linguistic features.

Proposed method

• Transcribed cow vocalizations into text using a Natural Language Processing (NLP) model to extract linguistic content.
• Extracted key acoustic features including frequency, duration, and intensity from audio recordings.
• Fused linguistic and acoustic data using a custom-developed ontology to create a comprehensive representation of vocalizations.
• Applied machine learning models—Random Forest, Support Vector Machine, and Recurrent Neural Networks—to classify vocalizations based on emotional states.
• Used sentiment analysis and contextual assessment to enrich the linguistic component of the fused data.
• Optimized models to handle data quality issues and computational demands typical in on-farm settings.

Experimental results

Research questions

• RQ1 Can acoustic and linguistic features be effectively fused to decode dairy cow vocalizations with improved accuracy?
• RQ2 How do specific acoustic features such as frequency and intensity correlate with emotional states like distress or contentment?
• RQ3 To what extent does incorporating transcribed linguistic content enhance the classification of cow vocalizations compared to acoustic features alone?
• RQ4 Which machine learning models perform best in classifying cow vocalizations under real-world farm conditions?
• RQ5 What emotional indicators can be reliably extracted from fused multi-modal data to support animal welfare assessment?

Key findings

• The fusion of acoustic and linguistic data significantly improved the classification of cow vocalizations into emotional states compared to single-modality approaches.
• High-frequency calls were consistently associated with distress or arousal, while low-frequency calls were linked to contentment or calmness.
• Specific acoustic features such as elevated frequency and distinct sound spectrum patterns were identified as reliable indicators of emotional distress.
• The Random Forest and Recurrent Neural Network models demonstrated superior performance in handling complex, noisy farm data and achieving high classification accuracy.
• Sentiment analysis of transcribed vocalizations revealed consistent emotional patterns across 20 individual cows, supporting the reliability of the fusion framework.
• The custom ontology enabled structured representation and categorization of vocalizations, enhancing interpretability and scalability of the system.