[Paper Review] CattleFace-RGBT: RGB-T Cattle Facial Landmark Benchmark

Ethan Coffman, Reagan Clark+6 more|arXiv (Cornell University)|Jun 5, 2024

Textile materials and evaluationsMaterials Science3 citations

TL;DR

This paper introduces CattleFace-RGBT, the first RGB-T (RGB and thermal) cattle facial landmark dataset with 2,300 image pairs and 13 annotated keypoint landmarks on facial features like eyes, ears, muzzle, and nostrils. Using an AI-assisted semi-automatic annotation pipeline that transfers knowledge from RGB to thermal images, the authors benchmark state-of-the-art models, establishing ResNet101 and Swin-B as strong baselines for future research in cattle welfare monitoring via multimodal facial analysis.

ABSTRACT

To address this challenge, we introduce CattleFace-RGBT, a RGB-T Cattle Facial Landmark dataset consisting of 2,300 RGB-T image pairs, a total of 4,600 images. Creating a landmark dataset is time-consuming, but AI-assisted annotation can help. However, applying AI to thermal images is challenging due to suboptimal results from direct thermal training and infeasible RGB-thermal alignment due to different camera views. Therefore, we opt to transfer models trained on RGB to thermal images and refine them using our AI-assisted annotation tool following a semi-automatic annotation approach. Accurately localizing facial key points on both RGB and thermal images enables us to not only discern the cattle's respiratory signs but also measure temperatures to assess the animal's thermal state. To the best of our knowledge, this is the first dataset for the cattle facial landmark on RGB-T images. We conduct benchmarking of the CattleFace-RGBT dataset across various backbone architectures, with the objective of establishing baselines for future research, analysis, and comparison. The dataset and models are at https://github.com/UARK-AICV/CattleFace-RGBT-benchmark

Motivation & Objective

• To address the lack of multimodal datasets combining RGB and thermal imaging for cattle facial landmark detection.
• To enable accurate, automated assessment of cattle health through facial keypoint localization and temperature measurement.
• To overcome challenges in thermal image annotation by using an AI-assisted transfer learning approach from RGB to thermal data.
• To establish performance baselines for keypoint detection models on a novel, real-world cattle dataset.
• To advance AI-driven cattle welfare monitoring by integrating facial expression and thermal state analysis.

Proposed method

• The dataset comprises 2,300 paired RGB and thermal images collected from a farm, with 13 facial landmarks annotated on key features including eyes, ears, muzzle, nostrils, and mouth.
• An in-house C# annotation tool enables precise manual correction of AI-generated landmarks, ensuring high annotation accuracy.
• A semi-automatic annotation pipeline transfers initial keypoint predictions from RGB-trained models to thermal images, followed by human-in-the-loop refinement.
• The method avoids direct thermal model training due to suboptimal performance and circumvents alignment issues from differing camera views.
• Backbone networks including ResNet50, ResNet101, ViT-B, and Swin-B are evaluated using the Detectron2 framework with standard AP, AP50, and AP75 metrics.
• Training uses a 70/30 split for training and testing, with a batch size of 16 and AdamW optimizer at a learning rate of 1e-6.

Figure 1 : Example of RGB and thermal pair image with 13 annotated keypoints annotated in our CattleFace-RGBT dataset.

Experimental results

Research questions

• RQ1 How effective are existing keypoint detection models on a novel RGB-T cattle facial landmark dataset?
• RQ2 Can knowledge from RGB-based models be effectively transferred to thermal images for accurate landmark annotation?
• RQ3 What are the performance differences between CNN-based and Transformer-based backbones on thermal cattle facial data?
• RQ4 How does the annotation quality and efficiency compare when using AI-assisted transfer learning versus manual-only labeling?
• RQ5 What are the baseline performance metrics for keypoint detection on multimodal cattle facial data?

Key findings

• ResNet101 achieved the highest overall performance on RGB images with 76.12 AP for bounding box detection and 94.37 AP for keypoint detection.
• On thermal images, ResNet101 achieved the highest scores across all metrics, including 72.30 AP for bounding boxes and 64.60 AP for keypoint detection.
• Swin-B achieved the highest AP50 and AP75 for bounding box detection on RGB images (100.00 and 100.00, respectively), indicating strong performance at high IoU thresholds.
• Thermal image performance was significantly lower than RGB, with AP scores dropping by up to 20–30 points, likely due to lower contrast and texture in thermal data.
• All models achieved 100.00 AP50 for keypoint detection on RGB images, indicating excellent performance at the 50% IoU threshold.
• The AI-assisted annotation pipeline enabled efficient and accurate landmark annotation despite challenges in thermal image quality and camera view misalignment.

Figure 2 : Examples of undesired images.