[Paper Review] Animal Behavior Analysis Methods Using Deep Learning: A Survey

Edoardo Fazzari, Donato Romano+2 more|arXiv (Cornell University)|May 22, 2024

Identification and Quantification in Food6 citations

TL;DR

A survey of deep learning architectures and strategies for animal behavior analysis across audio, visual, and audiovisual data, including pose estimation and non-pose methods, datasets, and future directions.

ABSTRACT

Animal behavior serves as a reliable indicator of the adaptation of organisms to their environment and their overall well-being. Through rigorous observation of animal actions and interactions, researchers and observers can glean valuable insights into diverse facets of their lives, encompassing health, social dynamics, ecological relationships, and neuroethological dimensions. Although state-of-the-art deep learning models have demonstrated remarkable accuracy in classifying various forms of animal data, their adoption in animal behavior studies remains limited. This survey article endeavors to comprehensively explore deep learning architectures and strategies applied to the identification of animal behavior, spanning auditory, visual, and audiovisual methodologies. Furthermore, the manuscript scrutinizes extant animal behavior datasets, offering a detailed examination of the principal challenges confronting this research domain. The article culminates in a comprehensive discussion of key research directions within deep learning that hold potential for advancing the field of animal behavior studies.

Motivation & Objective

• Explain why deep learning is transformative for animal behavior study and identify key limitations.
• Categorize deep learning approaches into pose estimation and non-pose estimation for behavior analysis.
• Compile publicly available datasets and analyze their coverage across species and contexts.
• Discuss challenges in data collection, annotation, and deployment to guide future research.

Proposed method

• Systematic literature review of 2020–2023 articles from Google Scholar, IEEE Xplore, and Springer.
• Classification of methods into pose estimation (APE) and non-pose estimation approaches.
• Analysis of datasets, species focus, and author disciplinary backgrounds.
• Discussion of advantages, limitations, and deployment considerations of DL in the field.
• Synthesis of emerging architectures (e.g., DLC, LEAP, SLEAP, optiFlex, SemiMultiPose) and their impact on behavior analysis.

Figure 1. a) illustrates a histogram depicting the distribution of research articles per year, focusing exclusively on papers obtained and cataloged during the initial scavenging phase. b) presents a pie chart detailing the variety of animals utilized in behavioral studies leveraging deep learning t

Experimental results

Research questions

• RQ1 RQ1 Which animal species are less considered and why in deep learning-based behavior analysis?
• RQ2 RQ2 What deep learning methods have been used in the literature for animal behavior analysis?
• RQ3 RQ3 What are the differences between human and animal behavior analysis in this context?
• RQ4 RQ4 What DL strategies exist that could enhance animal behavior analysis but are not yet exploited?

Key findings

• The survey analyzes 151 relevant articles from 2020–2023 with a focus on livestock, mice, and neuroscience-related studies.
• Pose estimation methods (DLC, LEAP, SLEAP) are central to many studies, with evolving architectures for multi-animal scenarios.
• Emerging DL approaches include OptiFlex, SemiMultiPose, Lightning Pose, and transformer-based or semi-supervised methods.
• Unsup ervised and self-supervised techniques are increasingly used to reduce labeling effort and improve generalization.
• There is a trend toward non-invasive vision-based methods and fusion of multiple sensors to mitigate noise and stress-induced data.
• The article highlights methodological and annotation challenges, cross-domain data requirements, and the need for diverse species coverage.

Figure 2. a) shows the architecture of LEAP (Pereira et al . , 2019 ) ; b) the one exploited in T-LEAP (Russello et al . , 2022 )