[Paper Review] Teaching Where to Look: Attention Similarity Knowledge Distillation for Low Resolution Face Recognition

Sungho Shin, Joosoon Lee+3 more|arXiv (Cornell University)|Sep 29, 2022

Face recognition and analysis4 citations

TL;DR

This paper proposes Attention Similarity Knowledge Distillation (A-SKD), a method that transfers attention maps from a high-resolution (HR) teacher network to a low-resolution (LR) student network using cosine similarity to improve LR face recognition. The approach significantly boosts performance across multiple LR benchmarks, outperforming state-of-the-art methods by guiding the student to focus on discriminative facial regions learned from HR data.

ABSTRACT

Deep learning has achieved outstanding performance for face recognition benchmarks, but performance reduces significantly for low resolution (LR) images. We propose an attention similarity knowledge distillation approach, which transfers attention maps obtained from a high resolution (HR) network as a teacher into an LR network as a student to boost LR recognition performance. Inspired by humans being able to approximate an object's region from an LR image based on prior knowledge obtained from HR images, we designed the knowledge distillation loss using the cosine similarity to make the student network's attention resemble the teacher network's attention. Experiments on various LR face related benchmarks confirmed the proposed method generally improved recognition performances on LR settings, outperforming state-of-the-art results by simply transferring well-constructed attention maps. The code and pretrained models are publicly available in the https://github.com/gist-ailab/teaching-where-to-look.

Motivation & Objective

• Address the significant performance drop in face recognition when applying HR-trained models to low-resolution (LR) images.
• Overcome spatial information loss in LR images by leveraging prior knowledge from HR data.
• Improve LR recognition by transferring attention maps—representing where to focus—rather than raw features or logits.
• Demonstrate generalization of the method beyond face recognition to face detection and general object classification.
• Develop a lightweight, efficient knowledge distillation framework that avoids the high cost of super-resolution

Proposed method

• Train a high-resolution (HR) teacher network to generate detailed, informative attention maps using CBAM modules.
• Use cosine similarity between the attention maps of the HR teacher and LR student networks as the knowledge distillation loss.
• Optimize the student network to align its attention maps with those of the teacher, emphasizing key facial regions like eyes, nose, and mouth.
• Integrate the A-SKD loss with standard logit distillation for improved performance in both recognition and detection tasks.
• Apply the method to ResNet18 with CBAM for generalization to object classification and face detection.
• Use data augmentation and down-sampling (4×, 16×, 32×) to simulate LR conditions across benchmarks

Experimental results

Research questions

• RQ1 Can attention maps from a high-resolution (HR) network serve as effective prior knowledge for low-resolution (LR) face recognition?
• RQ2 Does transferring attention maps via cosine similarity improve LR recognition performance compared to feature-based or logit-based distillation?
• RQ3 How does A-SKD generalize beyond face recognition to other LR vision tasks such as face detection and object classification?
• RQ4 Is the attention similarity loss more effective than existing knowledge distillation methods like AT and RKD in low-resolution settings?
• RQ5 Can A-SKD improve detection of small faces in surveillance-like LR images?

Key findings

• A-SKD achieved 66.52% top-1 accuracy on 4× down-sampled ImageNet, outperforming AT (65.79%) and RKD (65.95%) when combined with logit distillation.
• On WIDER FACE, A-SKD improved mAP by 15.72% on easy, 14.15% on medium, and 33.98% on hard face detection tasks at 16× down-sampling.
• At 32× down-sampling, A-SKD increased mAP by 7.54% on easy, 12.59% on medium, and 14.43% on hard faces, demonstrating robustness to extreme resolution loss.
• Qualitative results showed that A-SKD significantly enhanced detection of small faces in LR images, as visualized in Figure 6.
• Ablation studies confirmed that attention maps are the most effective form of prior knowledge for LR recognition, and cosine similarity is the optimal transfer mechanism.
• The method generalizes well beyond face recognition, showing consistent gains in general object classification and face detection tasks