[Paper Review] CKD-TransBTS: Clinical Knowledge-Driven Hybrid Transformer with Modality-Correlated Cross-Attention for Brain Tumor Segmentation

Jianwei Lin, Jiatai Lin+13 more|arXiv (Cornell University)|Jul 15, 2022

Brain Tumor Detection and Classification4 citations

TL;DR

This paper proposes CKD-TransBTS, a clinical knowledge-driven hybrid Transformer model for brain tumor segmentation that reorganizes MRI modalities into biologically meaningful pairs (T1/T1Gd and T2/T2FLAIR) based on radiological diagnosis principles. By introducing a modality-correlated cross-attention block (MCCA) and a Trans&CNN Feature Calibration block (TCFC), the model achieves state-of-the-art performance on BraTS 2021, with a mean Dice score of 0.9066 and HD95 of 6.22, significantly improving boundary precision and reducing false positives.

ABSTRACT

Brain tumor segmentation (BTS) in magnetic resonance image (MRI) is crucial for brain tumor diagnosis, cancer management and research purposes. With the great success of the ten-year BraTS challenges as well as the advances of CNN and Transformer algorithms, a lot of outstanding BTS models have been proposed to tackle the difficulties of BTS in different technical aspects. However, existing studies hardly consider how to fuse the multi-modality images in a reasonable manner. In this paper, we leverage the clinical knowledge of how radiologists diagnose brain tumors from multiple MRI modalities and propose a clinical knowledge-driven brain tumor segmentation model, called CKD-TransBTS. Instead of directly concatenating all the modalities, we re-organize the input modalities by separating them into two groups according to the imaging principle of MRI. A dual-branch hybrid encoder with the proposed modality-correlated cross-attention block (MCCA) is designed to extract the multi-modality image features. The proposed model inherits the strengths from both Transformer and CNN with the local feature representation ability for precise lesion boundaries and long-range feature extraction for 3D volumetric images. To bridge the gap between Transformer and CNN features, we propose a Trans&CNN Feature Calibration block (TCFC) in the decoder. We compare the proposed model with five CNN-based models and six transformer-based models on the BraTS 2021 challenge dataset. Extensive experiments demonstrate that the proposed model achieves state-of-the-art brain tumor segmentation performance compared with all the competitors.

Motivation & Objective

• To address the limitation in existing brain tumor segmentation models that treat multi-modal MRI fusion as a generic feature concatenation without clinical rationale.
• To incorporate radiological diagnostic knowledge—specifically, how clinicians compare T1/T1Gd for enhancing tumor and T2/T2FLAIR for edema—into the model architecture.
• To improve segmentation accuracy, especially for lesion boundaries and false positive suppression, by designing a modality-aware fusion mechanism.
• To bridge the representation gap between CNN and Transformer features through a novel feature calibration block in the decoder.
• To demonstrate that integrating clinical prior knowledge into deep learning architecture enhances model generalization and performance beyond label supervision.

Proposed method

• The input MRI modalities are re-grouped into two pairs: (T1, T1Gd) and (T2, T2FLAIR), based on their clinical imaging principles and diagnostic relevance.
• A dual-branch hybrid encoder is designed, where each branch processes one modality pair using a combination of CNN and Transformer blocks to capture local and long-range features.
• The Modality-Correlated Cross-Attention (MCCA) block enables cross-attention between paired modalities, learning modality-specific correlations to improve feature representation.
• The Trans&CNN Feature Calibration (TCFC) block is introduced in the decoder to align and calibrate features from the Transformer and CNN branches, reducing domain shift and bias.
• The model is trained end-to-end on the BraTS 2021 dataset using standard metrics: Dice score and Hausdorff distance (HD95).
• Ablation studies compare different fusion strategies: input-level, feature-level, and the proposed modality-pairing strategy with MCCA.

Experimental results

Research questions

• RQ1 Can reorganizing MRI modalities into clinically meaningful pairs improve brain tumor segmentation performance?
• RQ2 Does a modality-correlated cross-attention mechanism outperform conventional feature-level or input-level fusion in multi-modal MRI segmentation?
• RQ3 Can a hybrid Transformer-CNN architecture with a dedicated feature calibration block improve boundary precision and reduce false positives?
• RQ4 To what extent does incorporating clinical diagnostic knowledge enhance model generalization beyond supervised learning?
• RQ5 How does the proposed model compare to state-of-the-art methods on the BraTS 2021 benchmark?

Key findings

• The proposed CKD-TransBTS model achieves a mean Dice score of 0.9066 on the BraTS 2021 test set, outperforming all compared CNN-based and Transformer-based models.
• The model achieves an HD95 distance of 6.22, indicating superior boundary precision, especially for the enhancing tumor (ET) with HD95 of 5.93.
• The ablation study shows that grouping T1 and T1Gd as a pair yields the best performance, with a Dice score of 0.8850 for ET, demonstrating the value of clinical knowledge in fusion design.
• The MCCA block significantly improves segmentation accuracy over conventional fusion strategies, reducing HD95 by up to 2.5 mm compared to input-level fusion.
• Qualitative results confirm that the model detects small lesions and suppresses false positives more effectively than baselines, particularly when comparing T1 and T1Gd for enhancing tumor.
• The Trans&CNN Feature Calibration (TCFC) block effectively reduces the representation gap between Transformer and CNN features, improving feature alignment and segmentation consistency.