[Paper Review] AP-10K: A Benchmark for Animal Pose Estimation in the Wild

Hang Yu, Yufei Xu+4 more|arXiv (Cornell University)|Aug 28, 2021

Human Pose and Action Recognition37 references35 citations

TL;DR

AP-10K is a large-scale mammal pose-estimation benchmark with 10,015 labeled images across 23 families and 54 species, evaluating supervised learning, cross-domain transfer from human pose estimation, and intra-/inter-family generalization.

ABSTRACT

Accurate animal pose estimation is an essential step towards understanding animal behavior, and can potentially benefit many downstream applications, such as wildlife conservation. Previous works only focus on specific animals while ignoring the diversity of animal species, limiting the generalization ability. In this paper, we propose AP-10K, the first large-scale benchmark for mammal animal pose estimation, to facilitate the research in animal pose estimation. AP-10K consists of 10,015 images collected and filtered from 23 animal families and 54 species following the taxonomic rank and high-quality keypoint annotations labeled and checked manually. Based on AP-10K, we benchmark representative pose estimation models on the following three tracks: (1) supervised learning for animal pose estimation, (2) cross-domain transfer learning from human pose estimation to animal pose estimation, and (3) intra- and inter-family domain generalization for unseen animals. The experimental results provide sound empirical evidence on the superiority of learning from diverse animals species in terms of both accuracy and generalization ability. It opens new directions for facilitating future research in animal pose estimation. AP-10k is publicly available at https://github.com/AlexTheBad/AP10K.

Motivation & Objective

• Address the lack of a large-scale, diverse animal pose dataset for generalization to unseen species.
• Provide a taxonomically organized benchmark to study intra- and inter-family generalization.
• Benchmark representative pose-estimation models across multiple tracks to assess pretraining effects and transfer learning.
• Enable semi-supervised and self-supervised learning exploration using auxiliary unlabeled images labeled by family/species.

Proposed method

• Collect and curate a large-scale, taxonomically organized animal image dataset from public sources.
• Annotate 10,015 images with 17 keypoints per animal and 8 background types in COCO-style format, with multi-round quality checks.
• Organize data by family and species to enable intra- and inter-family evaluation.
• Evaluate representative pose-estimation models on three tracks: supervised learning (SL), cross-domain transfer learning from human pose estimation (CD-TL), and intra-/inter-family domain generalization (DG).
• Investigate effects of pretraining (ImageNet, COCO) and training schedule on performance; analyze cross-domain transfer gaps and few-/transfer-learning scenarios.

Experimental results

Research questions

• RQ1 How do representative human pose models perform on animal pose estimation on AP-10K?
• RQ2 Does training on diverse animal species improve accuracy and generalization to unseen animals?
• RQ3 What is the impact of pretraining (ImageNet or COCO human pose) on animal pose estimation with a large and diverse dataset?
• RQ4 How do intra-family and inter-family generalization behave when training on some species and testing on others?
• RQ5 Can cross-domain transfer from human pose estimation to animal pose estimation compensate for limited animal data, and how does training schedule affect this?

Key findings

• AP-10K contains 10,015 labeled images across 23 families and 54 species, with about 50k unlabeled images for semi/self-supervised use.
• 17 animal keypoints are defined per instance, with 8 background categories, annotated in COCO format and carefully checked by 13 annotators.
• Pretraining on ImageNet improves SL track performance, but longer training schedules can reduce this gap when training from scratch.
• Cross-domain transfer from human pose estimation to animal pose estimation helps when given longer finetuning (e.g., approaching or surpassing ImageNet pretraining in some settings).
• Intra- and inter-family generalization show that training on multiple species enhances feature representation, with varied performance when testing on unseen species across families; data from more diverse species generally improves generalization.
• The dataset exhibits a long-tail distribution across families and species, enabling few-shot learning studies and challenging generalization tasks.
• Cross-family transfer and few-shot experiments demonstrate that even small amounts of fine-tuning data can significantly boost performance on new species.