[Paper Review] Towards Self-Supervision for Video Identification of Individual Holstein-Friesian Cattle: The Cows2021 Dataset

Jing Gao, Tilo Burghardt+3 more|arXiv (Cornell University)|May 5, 2021

Food Supply Chain Traceability18 citations

TL;DR

This paper introduces the Cows2021 dataset, the largest identity-annotated Holstein-Friesian cattle dataset to date, and proposes a self-supervision framework for video-based individual cattle identification. Using temporal coat pattern consistency across videos, the method employs tracking-by-detection, rotation-normalized tracklets, data augmentation, and frame-triplet contrastive learning to learn a metric embedding space, followed by Gaussian Mixture Model clustering to produce a classifier achieving 57.0% Top-1 and 76.9% Top-4 accuracy without any manual identity labeling during training.

ABSTRACT

In this paper we publish the largest identity-annotated Holstein-Friesian cattle dataset Cows2021 and a first self-supervision framework for video identification of individual animals. The dataset contains 10,402 RGB images with labels for localisation and identity as well as 301 videos from the same herd. The data shows top-down in-barn imagery, which captures the breed's individually distinctive black and white coat pattern. Motivated by the labelling burden involved in constructing visual cattle identification systems, we propose exploiting the temporal coat pattern appearance across videos as a self-supervision signal for animal identity learning. Using an individual-agnostic cattle detector that yields oriented bounding-boxes, rotation-normalised tracklets of individuals are formed via tracking-by-detection and enriched via augmentations. This produces a `positive' sample set per tracklet, which is paired against a `negative' set sampled from random cattle of other videos. Frame-triplet contrastive learning is then employed to construct a metric latent space. The fitting of a Gaussian Mixture Model to this space yields a cattle identity classifier. Results show an accuracy of Top-1 57.0% and Top-4: 76.9% and an Adjusted Rand Index: 0.53 compared to the ground truth. Whilst supervised training surpasses this benchmark by a large margin, we conclude that self-supervision can nevertheless play a highly effective role in speeding up labelling efforts when initially constructing supervision information. We provide all data and full source code alongside an analysis and evaluation of the system.

Motivation & Objective

• To reduce the high labeling burden in training visual cattle identification systems by leveraging self-supervision.
• To create and release the largest identity-annotated dataset of Holstein-Friesian cattle for research.
• To develop a method that uses temporal coat pattern consistency across videos as a self-supervision signal for animal identity learning.
• To evaluate the feasibility of self-supervised representation learning for cattle biometric identification in real-world video settings.
• To provide a toolchain that can accelerate human annotation in production-level cattle ID systems.

Proposed method

• An individual-agnostic cattle detector generates oriented bounding boxes from RGB video frames, which are used to localize cattle in top-down barn imagery.
• Tracking-by-detection forms tracklets from detected bounding boxes, which are then normalized for scale and orientation to create consistent visual representations.
• Data augmentation is applied to enrich the positive sample set within each tracklet, enhancing robustness to appearance variation.
• Frame-triplet contrastive learning is used to train a metric embedding space, where each anchor is a frame from a tracklet and positives are other frames from the same tracklet, while negatives are sampled from different videos.
• A Gaussian Mixture Model (GMM) is fitted to the learned latent space to cluster embeddings and assign identity labels, treating each cluster as a potential identity.
• The system is evaluated using Top-N accuracy and Adjusted Rand Index (ARI), with identity assignments derived from cluster-overlap analysis.

Experimental results

Research questions

• RQ1 Can temporal consistency in coat pattern appearance across videos serve as an effective self-supervision signal for cattle identity learning?
• RQ2 To what extent can self-supervised representation learning reduce the need for manual annotation in cattle identification systems?
• RQ3 How well does a contrastive learning framework with GMM-based clustering perform on a large-scale cattle re-identification benchmark with no identity labels during training?
• RQ4 Can the proposed method produce a confidence-ranked list of candidate identities that aids human annotation in real-world deployment?
• RQ5 What is the performance ceiling of self-supervised learning for biometric cattle identification compared to fully supervised approaches?

Key findings

• The proposed self-supervised framework achieved a Top-1 accuracy of 57.0% and Top-4 accuracy of 76.9% on the test set, demonstrating strong feasibility for reducing labeling effort.
• The Adjusted Rand Index (ARI) of 0.53 indicates a meaningful structural similarity between the GMM clustering and ground-truth identity assignments.
• The method enables the generation of a confidence-ranked list of candidate identities, with a 76.9% chance of including the correct identity in the top four suggestions.
• Despite not using any manual identity annotations during training, the system produces a usable classifier that can significantly accelerate human labeling in production pipelines.
• The results show that self-supervision can effectively bootstrap visual cattle identification systems, especially when combined with subsequent open-set fine-tuning.
• The Cows2021 dataset, with 10,402 annotated images and 301 videos from 182 identifiable Holstein-Friesian cattle, is the largest such public dataset to date.