[Paper Review] Deep Learning Models for Automated Classification of Dog Emotional States from Facial Expressions

Tali Boneh-Shitrit, Shir Amir+5 more|arXiv (Cornell University)|Jun 11, 2022

Human-Animal Interaction Studies4 citations

TL;DR

This study proposes a deep learning framework for automated classification of dog emotional states—specifically positive anticipation and frustration—using facial expressions from a controlled experimental dataset. Using self-supervised pretraining, DINO-ViT outperformed ResNet and other backbones, demonstrating superior accuracy and localized attention on key facial regions relevant to emotional expression.

ABSTRACT

Similarly to humans, facial expressions in animals are closely linked with emotional states. However, in contrast to the human domain, automated recognition of emotional states from facial expressions in animals is underexplored, mainly due to difficulties in data collection and establishment of ground truth concerning emotional states of non-verbal users. We apply recent deep learning techniques to classify (positive) anticipation and (negative) frustration of dogs on a dataset collected in a controlled experimental setting. We explore the suitability of different backbones (e.g. ResNet, ViT) under different supervisions to this task, and find that features of a self-supervised pretrained ViT (DINO-ViT) are superior to the other alternatives. To the best of our knowledge, this work is the first to address the task of automatic classification of canine emotions on data acquired in a controlled experiment.

Motivation & Objective

• To develop an automated deep learning system for classifying canine emotional states from facial expressions, focusing on positive anticipation and frustration.
• To evaluate the performance of different deep learning backbones—ResNet and Vision Transformer (ViT)—on canine emotion recognition.
• To compare supervised and self-supervised pretraining strategies (e.g., ImageNet vs. DINO) for improving model generalization and performance.
• To assess model interpretability using activation maps and link learned features to behavioral concepts such as DogFACS action units.
• To establish a reliable, objective baseline for canine affective computing using a rigorously collected, controlled dataset.

Proposed method

• Utilized a controlled experimental dataset with Labrador Retrievers to induce positive anticipation and frustration, ensuring consistent and reliable ground truth.
• Trained multiple deep learning models using different backbones: ResNet and Vision Transformer (ViT), with both supervised (ImageNet) and self-supervised (DINO) pretraining.
• Applied Eigen-CAM for visualization of activation maps to interpret model attention and assess localization on facial regions such as eyes, ears, and nose.
• Evaluated model performance using standard classification metrics (e.g., accuracy, F1-score) on the balanced test set.
• Leveraged DINO-ViT’s self-supervised pretraining to extract semantically rich features sensitive to object parts, enhancing performance on subtle facial movements.
• Mapped model attention to DogFACS action units to validate alignment with behavioral science concepts.

Experimental results

Research questions

• RQ1 Can deep learning models accurately classify canine emotional states—specifically positive anticipation and frustration—using facial expressions from a controlled experiment?
• RQ2 How do different backbone architectures (ResNet vs. ViT) compare in performance for canine emotion recognition on this task?
• RQ3 Does self-supervised pretraining (DINO) outperform supervised pretraining (ImageNet) for this low-resource, high-variability animal emotion recognition task?
• RQ4 To what extent can model activation maps localize to biologically relevant facial regions, and how does this relate to known action units in DogFACS?
• RQ5 Can explainability techniques like Eigen-CAM provide interpretable insights into model predictions and failures, supporting scientific validation?

Key findings

• The DINO-ViT model achieved the highest classification accuracy among all tested backbones, outperforming both ResNet and ImageNet-pretrained ViT.
• Self-supervised pretraining with DINO led to significantly better performance than supervised ImageNet pretraining, especially in capturing subtle facial cues.
• ViT-based models demonstrated superior localization of attention on salient facial regions (eyes, ears, nose) compared to ResNet, which showed broader and less focused activation patterns.
• Eigen-CAM visualization revealed that DINO-ViT activated multiple key facial regions simultaneously, aligning with human-annotated DogFACS action units.
• The DINO-ViT model showed consistent attention on behaviorally relevant features, suggesting its features are sensitive to object parts and facial movements linked to emotion.
• Model predictions were interpretable via Eigen-CAM, even when incorrect, due to its class-independent nature, enabling analysis of failure modes and feature relevance.