[Paper Review] Interspecies Knowledge Transfer for Facial Keypoint Detection

Maheen Rashid, Xiuye Gu+1 more|arXiv (Cornell University)|Apr 13, 2017

Face recognition and analysis45 references4 citations

TL;DR

This paper proposes a novel method for facial keypoint detection in animals by transferring knowledge from human facial keypoint detection through shape-aware warping. It first warps animal faces to resemble human faces using a thin-plate spline network trained on nearest human face neighbors, then finetunes a pre-trained human keypoint detector on the warped animal data, achieving state-of-the-art performance, especially with limited training data.

ABSTRACT

We present a method for localizing facial keypoints on animals by transferring knowledge gained from human faces. Instead of directly finetuning a network trained to detect keypoints on human faces to animal faces (which is sub-optimal since human and animal faces can look quite different), we propose to first adapt the animal images to the pre-trained human detection network by correcting for the differences in animal and human face shape. We first find the nearest human neighbors for each animal image using an unsupervised shape matching method. We use these matches to train a thin plate spline warping network to warp each animal face to look more human-like. The warping network is then jointly finetuned with a pre-trained human facial keypoint detection network using an animal dataset. We demonstrate state-of-the-art results on both horse and sheep facial keypoint detection, and significant improvement over simple finetuning, especially when training data is scarce. Additionally, we present a new dataset with 3717 images with horse face and facial keypoint annotations.

Motivation & Objective

• To address the lack of large-scale annotated datasets for animal facial keypoint detection, especially for species like horses and sheep.
• To improve keypoint detection accuracy in animals by transferring knowledge from human facial keypoint detection, which benefits from abundant annotated data.
• To overcome the suboptimal performance of direct fine-tuning from human to animal faces due to structural and appearance differences.
• To develop a method that explicitly corrects for interspecies shape differences before finetuning, enabling better adaptation with limited data.
• To release a new, large-scale horse facial keypoint dataset to support future research in animal facial analysis.

Proposed method

• The method uses an unsupervised shape matching approach to find the nearest human face neighbors for each animal image based on keypoint configurations.
• A thin-plate spline (TPS) warping network is trained to deform each animal face to match the shape of its nearest human face neighbors, minimizing shape discrepancy.
• The warped animal images are used to finetune a pre-trained deep neural network for human facial keypoint detection, allowing the network to focus on appearance differences rather than shape differences.
• The warping network and keypoint detection network are jointly optimized during finetuning to improve alignment and detection accuracy.
• The system is evaluated on both horse and sheep datasets, with ablation studies on warping accuracy and neighbor count.
• A new dataset of 3,717 horse images with facial keypoint annotations is introduced to support benchmarking and future research.

Experimental results

Research questions

• RQ1 Can knowledge transfer from human facial keypoint detection be effectively adapted to animals despite significant shape and appearance differences?
• RQ2 Does warping animal faces to resemble human faces before finetuning improve keypoint detection performance compared to direct fine-tuning?
• RQ3 How does the performance of the proposed method scale with decreasing amounts of training data?
• RQ4 What is the impact of warping accuracy and the number of nearest human neighbors on detection performance?
• RQ5 Can a shape-aware warping strategy reduce the need for large-scale annotated animal datasets?

Key findings

• The proposed method achieves state-of-the-art performance on both horse and sheep facial keypoint detection, outperforming simple fine-tuning and TPS-based baselines.
• On the horse dataset, the method achieves a 6.72 percentage point lower failure rate than the TPS baseline and a 13.39 percentage point lower failure rate than simple fine-tuning when using only 500 training images.
• The performance gap between the proposed method and baselines widens as training data size decreases, demonstrating superior data efficiency.
• Using ground-truth warped images (GT Warp) as a baseline yields a failure rate of 7.76%, while the proposed method achieves 8.36%, showing that the warping network is effective but not perfect.
• The error rate decreases with more nearest neighbors (K) up to a point, but performance degrades with K > 10 due to noisy neighbor selection, indicating an optimal neighbor count exists.
• The method is robust and practical for small datasets, making it suitable for real-world applications where collecting large annotated animal datasets is infeasible.