[Paper Review] Vision-based Behavioral Recognition of Novelty Preference in Pigs

Aniket Shirke, Rebecca K. Golden+4 more|arXiv (Cornell University)|Jun 23, 2021

Human Pose and Action Recognition10 references4 citations

TL;DR

This paper introduces the Pig Novelty Preference Behavior (PNPB) dataset, a fully annotated video collection for automated behavioral recognition in pigs using deep learning. It evaluates LRCN, C3D, and TSM models on action recognition and NOR metric prediction, achieving 93% accuracy and 96% mean Average Precision, though challenges remain in regressing fine-grained metrics like investigation count and latency due to rapid pig movements.

ABSTRACT

Behavioral scoring of research data is crucial for extracting domain-specific metrics but is bottlenecked on the ability to analyze enormous volumes of information using human labor. Deep learning is widely viewed as a key advancement to relieve this bottleneck. We identify one such domain, where deep learning can be leveraged to alleviate the process of manual scoring. Novelty preference paradigms have been widely used to study recognition memory in pigs, but analysis of these videos requires human intervention. We introduce a subset of such videos in the form of the 'Pig Novelty Preference Behavior' (PNPB) dataset that is fully annotated with pig actions and keypoints. In order to demonstrate the application of state-of-the-art action recognition models on this dataset, we compare LRCN, C3D, and TSM on the basis of various analytical metrics and discuss common pitfalls of the models. Our methods achieve an accuracy of 93% and a mean Average Precision of 96% in estimating piglet behavior. We open-source our code and annotated dataset at https://github.com/AIFARMS/NOR-behavior-recognition

Motivation & Objective

• To address the bottleneck in manual video annotation for pig behavioral studies by developing an automated, deep learning-based solution.
• To create a fully annotated dataset—PNPB—featuring pig actions and keypoints for novelty preference testing.
• To evaluate state-of-the-art action recognition models (LRCN, C3D, TSM) on pig video data for both clip-level and video-level performance.
• To assess the models' ability to predict domain-specific NOR metrics (e.g., recognition index, investigation duration) with human-level accuracy.
• To identify technical limitations in current models when applied to rapid animal behaviors, especially for metrics involving timing and short-duration actions.

Proposed method

• The PNPB dataset contains 5.5-minute videos of piglets in a novelty preference arena, annotated with actions (e.g., investigate) and 2D keypoints.
• Three deep learning models—LRCN, C3D, and TSM—are trained and evaluated on clip-level action recognition using 30-frame clips.
• LRCN uses a ResNet-18 backbone for spatial feature extraction followed by an LSTM for temporal modeling.
• C3D employs 3D convolutions on video clips to learn spatio-temporal features, fine-tuned on the Sports1M pretraining.
• TSM uses a ResNet-18 backbone with a Temporal Shift Module to efficiently model temporal dynamics without RNNs.
• Video-level performance is evaluated using continuous action recognition metrics (TP, TN, O, U, F, M, I, D) to assess boundary alignment and error types.

Experimental results

Research questions

• RQ1 Can state-of-the-art action recognition models achieve human-level accuracy in classifying pig behaviors in novelty preference tasks?
• RQ2 To what extent can these models predict domain-specific NOR metrics such as recognition index, investigation duration, and latency?
• RQ3 What are the primary failure modes of deep learning models when applied to rapid, short-duration pig behaviors?
• RQ4 How do computational efficiency and model architecture (e.g., RNN vs. 3D-CNN vs. TSM) affect performance and inference speed?
• RQ5 Why do models fail to accurately regress metrics like investigation count and latency despite high clip-level accuracy?

Key findings

• The models achieve 93% accuracy and 96% mean Average Precision in classifying pig behaviors on short video clips.
• The TSM model offers the best trade-off between computational efficiency and performance, with 14.2 FPS on CPU and 2.1 GMACs.
• Continuous action recognition shows high true positive and negative rates (>90%), but severe errors (insertions/deletions) occur in ~3% of the time.
• The model predicts CD (total investigation duration) and RI (recognition index) with high R² (0.99 and 0.86, respectively), indicating strong regression performance.
• The model underestimates the number of investigations (N) and struggles with LF and LL metrics, showing low R² (0.58, 0.42, and -0.26, respectively).
• The primary challenge lies in detecting brief, high-speed interactions lasting less than a second, which are critical for accurate NOR metric computation.