[Paper Review] CVB: A Video Dataset of Cattle Visual Behaviors

Ali Zia, Renuka Sharma+8 more|arXiv (Cornell University)|May 26, 2023

Animal Behavior and Welfare Studies5 citations

TL;DR

This paper introduces CVB, a large-scale video dataset of 502 fifteen-second clips of grazing cattle in natural lighting, annotated with 11 visually perceptible behaviors. Using pre-trained YOLOv7 and Botsort models for initial detection and tracking, followed by expert-verified labeling in CVAT, the authors reduce annotation effort by 71%. The dataset enables accurate localization and recognition of common behaviors like grazing (73.96% accuracy) and resting-lying (58% accuracy) using the SlowFast model, with promising results on rare behaviors despite class imbalance.

ABSTRACT

Existing image/video datasets for cattle behavior recognition are mostly small, lack well-defined labels, or are collected in unrealistic controlled environments. This limits the utility of machine learning (ML) models learned from them. Therefore, we introduce a new dataset, called Cattle Visual Behaviors (CVB), that consists of 502 video clips, each fifteen seconds long, captured in natural lighting conditions, and annotated with eleven visually perceptible behaviors of grazing cattle. We use the Computer Vision Annotation Tool (CVAT) to collect our annotations. To make the procedure more efficient, we perform an initial detection and tracking of cattle in the videos using appropriate pre-trained models. The results are corrected by domain experts along with cattle behavior labeling in CVAT. The pre-hoc detection and tracking step significantly reduces the manual annotation time and effort. Moreover, we convert CVB to the atomic visual action (AVA) format and train and evaluate the popular SlowFast action recognition model on it. The associated preliminary results confirm that we can localize the cattle and recognize their frequently occurring behaviors with confidence. By creating and sharing CVB, our aim is to develop improved models capable of recognizing all important behaviors accurately and to assist other researchers and practitioners in developing and evaluating new ML models for cattle behavior classification using video data.

Motivation & Objective

• To address the lack of large, realistic, and well-annotated video datasets for cattle behavior recognition in natural environments.
• To reduce manual annotation effort by leveraging pre-trained object detection and tracking models (YOLOv7 and Botsort) before expert labeling.
• To create a dataset that supports robust machine learning models capable of recognizing both frequent and rare cattle behaviors under real-world conditions.
• To enable the development and evaluation of new deep learning models for animal behavior classification using video data.
• To support improved livestock management, welfare monitoring, and environmental sustainability through accurate behavior recognition.

Proposed method

• Captured 502 fifteen-second video clips of eight Angus cattle in natural lighting using high-resolution cameras at four corners of a pasture.
• Pre-processed videos by dividing them into fixed time segments and applying YOLOv7 for object detection and Botsort for multi-object tracking.
• Used the Computer Vision Annotation Tool (CVAT) to manually correct detections and label behaviors, with domain experts ensuring accuracy.
• Converted the final annotations into the Atomic Visual Action (AVA) format to support action recognition models that handle multiple actors and interactions.
• Trained and evaluated the SlowFast action recognition model on the AVA-formatted CVB dataset using an 80:20 training-testing split.
• Applied intersection over union (IoU) thresholding (>50%) to assess localization accuracy and reported confidence scores for behavior predictions.

Experimental results

Research questions

• RQ1 Can a large-scale video dataset of cattle in natural lighting with detailed behavioral annotations improve the performance of machine learning models for behavior recognition?
• RQ2 To what extent can pre-trained object detection and tracking models reduce the manual annotation burden in cattle behavior datasets?
• RQ3 How well can a state-of-the-art action recognition model (SlowFast) perform on a real-world cattle behavior dataset with long-tailed class distributions?
• RQ4 Can models trained on CVB successfully recognize both frequent behaviors (e.g., grazing) and rare behaviors (e.g., grooming, running) under challenging conditions like occlusion and variable lighting?
• RQ5 How effective is the AVA format in representing multi-actor behaviors and transitions between actions in cattle video data?

Key findings

• The CVB dataset contains 502 video clips, each 15 seconds long, with 225,900 annotated frames and 136,598 corrected bounding boxes, indicating that only 29% of initial detections required manual correction.
• Grazing is the most frequent behavior (39% of the time), followed by resting-lying (15%), resting-standing (12%), and hidden (10%), with all others occurring less than 5% of the time, reflecting a long-tailed distribution.
• The SlowFast model achieved a test accuracy of 73.96% for grazing and 58% for resting-lying when localization IoU exceeded 50%, with average confidence scores of 89% and 82%, respectively.
• Despite limited training data, the model showed reasonable performance on rare behaviors such as mild aggression and grooming, indicating potential for generalization to low-resource classes.
• The model successfully localized and recognized behaviors in complex scenes with occlusions and multiple cattle, including cases where multiple animals were partially hidden (Fig. 2b).
• The use of pre-trained detection and tracking models reduced annotation time and effort significantly, making large-scale data curation feasible and scalable.