[Paper Review] DATR: Domain-adaptive transformer for multi-domain landmark detection

Heqin Zhu, Qingsong Yao+1 more|arXiv (Cornell University)|Mar 12, 2022

Medical Imaging and Analysis5 citations

TL;DR

This paper proposes DATR, the first domain-adaptive transformer model for universal multi-domain anatomical landmark detection across head, hand, and chest X-rays. By integrating domain-adaptive self-attention and a lightweight guidance network with a Swin Transformer encoder, DATR achieves state-of-the-art performance, reducing mean registration error to 1.47px on mixed datasets and outperforming prior CNN-based models significantly.

ABSTRACT

Accurate anatomical landmark detection plays an increasingly vital role in medical image analysis. Although existing methods achieve satisfying performance, they are mostly based on CNN and specialized for a single domain say associated with a particular anatomical region. In this work, we propose a universal model for multi-domain landmark detection by taking advantage of transformer for modeling long dependencies and develop a domain-adaptive transformer model, named as DATR, which is trained on multiple mixed datasets from different anatomies and capable of detecting landmarks of any image from those anatomies. The proposed DATR exhibits three primary features: (i) It is the first universal model which introduces transformer as an encoder for multi-anatomy landmark detection; (ii) We design a domain-adaptive transformer for anatomy-aware landmark detection, which can be effectively extended to any other transformer network; (iii) Following previous studies, we employ a light-weighted guidance network, which encourages the transformer network to detect more accurate landmarks. We carry out experiments on three widely used X-ray datasets for landmark detection, which have 1,588 images and 62 landmarks in total, including three different anatomies (head, hand, and chest). Experimental results demonstrate that our proposed DATR achieves state-of-the-art performances by most metrics and behaves much better than any previous convolution-based models. The code will be released publicly.

Motivation & Objective

• To develop a universal model capable of detecting anatomical landmarks across multiple anatomical domains (e.g., head, hand, chest) without domain-specific retraining.
• To overcome the limitations of single-domain CNN-based models by leveraging transformers for long-range contextual modeling in multi-domain landmark detection.
• To design a domain-adaptive transformer block that enables shared and domain-specific feature learning across diverse anatomical regions.
• To improve detection accuracy through a lightweight guidance network that provides coarse but unambiguous landmark supervision.
• To validate the effectiveness of multi-domain pretraining and zero-shot transfer to unseen domains like pelvis.

Proposed method

• The model employs a transformer-based encoder built on Swin Transformer blocks, enhanced with domain-adaptive self-attention (MSA_Q^d) to model inter-domain relationships.
• A domain-adaptive transformer block (DAT) integrates domain-shared and domain-specific parameters, enabling shared knowledge extraction and domain-specific adaptation.
• The decoder uses domain-adaptive convolutions (DAC), replacing standard convolutions with channel-wise and point-wise operations to support multiple domains.
• A guidance network with five dilated convolutions and a 1×1 convolution generates a coarse heatmap to guide the main transformer network toward more accurate landmark predictions.
• The model is trained end-to-end on a mixed dataset of head, hand, and chest X-rays, with domain-specific parameters fine-tuned during inference on new domains.
• Transfer learning is evaluated by freezing domain-shared parameters and adding new domain-specific layers for a novel pelvis dataset.

Experimental results

Research questions

• RQ1 Can a transformer-based architecture achieve superior landmark detection performance across multiple anatomical domains compared to prior CNN-based models?
• RQ2 Does domain-adaptive attention in a transformer encoder improve feature learning for multi-domain landmark detection?
• RQ3 Can a single universal model trained on mixed datasets generalize effectively to unseen anatomical domains like pelvis?
• RQ4 How does multi-domain pretraining affect performance on individual domains compared to single-domain training?
• RQ5 To what extent does the guidance network enhance detection accuracy in the absence of strong supervision?

Key findings

• DATR achieves a mean registration error (MRE) of 1.47px on the combined head+hand+chest dataset, a significant improvement over the baseline transformer's 2.62px.
• On the head dataset, DATR achieves an SDR<3px of 78.07%, outperforming the single-domain baseline and demonstrating strong generalization.
• When fine-tuned on a novel pelvis dataset, the domain-adaptive transformer reduced MRE from 5.64px (baseline) to 4.12px, showing effective domain adaptation.
• Multi-domain training consistently improves performance across all domains, with the three-domain model achieving the best MRE and SDR scores on all test sets.
• The ablation study confirms that both domain-adaptive self-attention and the diagonal matrix adaptation (D^d) contribute significantly to performance, with the full model outperforming all variants.
• Visualization shows that predicted landmarks (red) closely align with annotated landmarks (green), especially on complex structures like the hand and chest.