[Paper Review] From Saliency to DINO: Saliency-guided Vision Transformer for Few-shot Keypoint Detection

Changsheng Lu, Hao Zhu+1 more|arXiv (Cornell University)|Apr 6, 2023

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes SalViT, a saliency-guided vision transformer that enhances few-shot keypoint detection by using masked self-attention and a morphology learner to focus on foreground regions and dynamically adapt receptive fields. The method achieves up to 10% higher PCK under severe occlusions and outperforms standard models by leveraging DINO as a lightweight saliency detector.

ABSTRACT

Unlike current deep keypoint detectors that are trained to recognize limited number of body parts, few-shot keypoint detection (FSKD) attempts to localize any keypoints, including novel or base keypoints, depending on the reference samples. FSKD requires the semantically meaningful relations for keypoint similarity learning to overcome the ubiquitous noise and ambiguous local patterns. One rescue comes with vision transformer (ViT) as it captures long-range relations well. However, ViT may model irrelevant features outside of the region of interest due to the global attention matrix, thus degrading similarity learning between support and query features. In this paper, we present a novel saliency-guided vision transformer, dubbed SalViT, for few-shot keypoint detection. Our SalViT enjoys a uniquely designed masked self-attention and a morphology learner, where the former introduces saliency map as a soft mask to constrain the self-attention on foregrounds, while the latter leverages the so-called power normalization to adjust morphology of saliency map, realizing ``dynamically changing receptive field''. Moreover, as salinecy detectors add computations, we show that attentive masks of DINO transformer can replace saliency. On top of SalViT, we also investigate i) transductive FSKD that enhances keypoint representations with unlabelled data and ii) FSKD under occlusions. We show that our model performs well on five public datasets and achieves ~10% PCK higher than the normally trained model under severe occlusions.

Motivation & Objective

• To address the challenge of noisy and ambiguous local patterns in few-shot keypoint detection (FSKD), especially under domain shifts and limited supervision.
• To improve similarity learning between support and query features by modeling long-range relations within salient foreground regions.
• To reduce computational overhead by replacing external saliency detectors with attentive features from DINO, which exhibit strong alignment with saliency maps.
• To enhance robustness to keypoint occlusions through a masking and alignment (MAA) strategy that improves generalization under partial supervision.

Proposed method

• Introduces a masked self-attention (M-SA) module that uses a saliency map as a soft mask to restrict self-attention computation to foreground patches only.
• Proposes a morphology learner (ML) that applies power normalization to the saliency map, enabling dynamic adjustment of the receptive field to better capture relevant context around keypoint candidates.
• Employs DINO's attention maps as a substitute for external saliency detectors, reducing inference cost while maintaining performance.
• Designs a transductive FSKD variant that enriches support keypoint prototypes using unlabeled data to improve representation quality in low-shot settings.
• Applies a masking and alignment (MAA) strategy during training, including RGB, saliency, and CNN feature masking, along with probability map alignment for occlusion robustness.
• Uses multiple alignment strategies—image-space (SimMIM), prediction-space (non-occluded loss), and feature-space (ℓ1, ℓ2, MMD)—to improve generalization under occlusion.

Experimental results

Research questions

• RQ1 Can saliency-guided attention improve few-shot keypoint detection by focusing self-attention on relevant foreground regions and suppressing background noise?
• RQ2 How effective is the morphology learner in enabling dynamic receptive field adaptation to enhance feature representation in few-shot keypoint detection?
• RQ3 Can DINO's attention maps serve as a lightweight, effective substitute for external saliency detectors without sacrificing performance?
• RQ4 To what extent does the proposed masking and alignment (MAA) strategy improve robustness to keypoint occlusions in few-shot settings?
• RQ5 Does transductive FSKD with unlabeled data improve keypoint prototype quality and generalization in low-shot scenarios?

Key findings

• The SalViT model achieves a 10% relative improvement in PCK (up to ~10% higher) on the Animal Pose dataset under 100% occlusion compared to a normally trained model.
• The proposed M-SA and ML components together yield a 3.23% absolute gain in harmonic score on the Animal Pose dataset in 1-shot transductive FSKD, compared to inductive baseline.
• Using DINO's attention maps as a saliency proxy reduces computational cost while maintaining performance, with minimal drop in accuracy compared to using external saliency models.
• The probability map alignment strategy in MAA outperforms other alignment methods, boosting PCK by up to 2% compared to masking-only and by 10% compared to the standard model under 100% occlusion.
• The ablation study confirms that masking RGB, saliency, and CNN features during training improves robustness, with the full MAA strategy (masking + probability alignment) achieving the highest performance under severe occlusion.
• Transductive FSKD with prototype refinement via unlabeled data improves performance by 4.97% on Animal Pose and 4.91% on AwA, demonstrating the benefit of data enrichment in low-shot regimes.