[Paper Review] Segmentation Enhanced Lameness Detection in Dairy Cows from RGB and Depth Video

Eric Arazo, Robin Aly+1 more|arXiv (Cornell University)|Jun 9, 2022

Animal Behavior and Welfare Studies4 citations

TL;DR

This paper proposes a segmentation-enhanced lameness detection system for dairy cows using RGB and depth video, leveraging pre-trained SlowFast networks on masked cow regions to improve accuracy. By replacing raw video input with binary segmentation masks, the method boosts side-view detection accuracy from 61.76% to 84.56%, demonstrating that focusing on cow body structure significantly enhances lameness classification performance.

ABSTRACT

Cow lameness is a severe condition that affects the life cycle and life quality of dairy cows and results in considerable economic losses. Early lameness detection helps farmers address illnesses early and avoid negative effects caused by the degeneration of cows' condition. We collected a dataset of short clips of cows passing through a hallway exiting a milking station and annotated the degree of lameness of the cows. This paper explores the resulting dataset and provides a detailed description of the data collection process. Additionally, we proposed a lameness detection method that leverages pre-trained neural networks to extract discriminative features from videos and assign a binary score to each cow indicating its condition: "healthy" or "lame." We improve this approach by forcing the model to focus on the structure of the cow, which we achieve by substituting the RGB videos with binary segmentation masks predicted with a trained segmentation model. This work aims to encourage research and provide insights into the applicability of computer vision models for cow lameness detection on farms.

Motivation & Objective

• Address the challenge of early lameness detection in dairy cows to reduce economic and welfare losses.
• Overcome subjectivity and expert dependency in manual lameness scoring by developing an automated, computer vision-based solution.
• Investigate the impact of camera position, input features, and class imbalance on lameness detection performance.
• Improve model robustness by forcing attention on cow body structure through segmentation masks.
• Provide a publicly available dataset and method to encourage further research in farm-based computer vision applications.

Proposed method

• Collected a proprietary dataset of 1,733 cow video clips from side and top views using Intel Realsense D435 cameras and RFID cow identification.
• Converted depth videos to RGB using HUE color space encoding for visual consistency with RGB input.
• Trained a semantic segmentation model to generate binary masks of cows, isolating the animal from background clutter.
• Applied SlowFast 3D CNNs to extract spatio-temporal features from video inputs, using both RGB and masked video streams.
• Concatenated features from slow and fast pathways (2304-dim total) for improved representation learning.
• Trained a three-layer classifier with ReLU and softmax activation to predict 'healthy' or 'lame' with Adam optimizer and cross-entropy loss.

Experimental results

Research questions

• RQ1 How does camera position (side vs. top view) affect lameness detection accuracy?
• RQ2 What is the impact of using raw RGB, depth, or segmentation-masked video inputs on classification performance?
• RQ3 How does applying segmentation masks to video inputs improve feature learning and model generalization?
• RQ4 To what extent does class imbalance (due to healthy cow dominance) affect model performance, and how can it be mitigated?
• RQ5 What is the computational cost of depth map preprocessing, and how does it affect real-time deployment on farms?

Key findings

• Side-view recordings achieved significantly higher accuracy (84.56%) than top-view (75.00%) when using segmentation masks over RGB input.
• Segmentation masks improved side-view accuracy by 22.8 percentage points (from 61.76% to 84.56%) compared to raw RGB video.
• For depth video input, masking improved side-view accuracy from 63.23% to 75.00%, but reduced performance in top-view settings.
• Recall for the 'lame' class increased from 27% to 80% on side views when using segmentation masks, indicating better detection of severe cases.
• Precision for the 'lame' class improved to 82% on side views with masked RGB input, showing reduced false positives.
• The HUE encoding of depth maps was computationally expensive and identified as a bottleneck for real-time farm deployment.