[Paper Review] TransBTS: Multimodal Brain Tumor Segmentation Using Transformer

Wenxuan Wang, Chen Chen+4 more|arXiv (Cornell University)|Mar 7, 2021

Advanced Neural Network ApplicationsComputer Science23 references123 citations

TL;DR

TransBTS introduces a 3D CNN encoder–Transformer architecture to model local and global features for multimodal brain tumor segmentation on 3D MRI, achieving competitive or superior results to state-of-the-art methods on BraTS 2019 and 2020.

ABSTRACT

Transformer, which can benefit from global (long-range) information modeling using self-attention mechanisms, has been successful in natural language processing and 2D image classification recently. However, both local and global features are crucial for dense prediction tasks, especially for 3D medical image segmentation. In this paper, we for the first time exploit Transformer in 3D CNN for MRI Brain Tumor Segmentation and propose a novel network named TransBTS based on the encoder-decoder structure. To capture the local 3D context information, the encoder first utilizes 3D CNN to extract the volumetric spatial feature maps. Meanwhile, the feature maps are reformed elaborately for tokens that are fed into Transformer for global feature modeling. The decoder leverages the features embedded by Transformer and performs progressive upsampling to predict the detailed segmentation map. Extensive experimental results on both BraTS 2019 and 2020 datasets show that TransBTS achieves comparable or higher results than previous state-of-the-art 3D methods for brain tumor segmentation on 3D MRI scans. The source code is available at https://github.com/Wenxuan-1119/TransBTS

Motivation & Objective

• Motivate modeling of both local 3D context and global long-range dependencies in volumetric MRI for segmentation.
• Propose a 3D CNN–Transformer encoder–decoder network (TransBTS) for brain tumor segmentation.
• Demonstrate effectiveness on BraTS 2019 and 2020 and analyze architecture choices via ablations.

Proposed method

• Use a 3D CNN encoder to extract compact local 3D feature maps from multimodal MRI.
• Flatten and project encoder features into Transformer-compatible tokens with added learnable position embeddings.
• Process tokens through a Transformer encoder to capture global long-range dependencies across spatial and depth dimensions.
• Map Transformer outputs back to a 4D feature map and apply a 3D CNN decoder with progressive upsampling and skip connections for full-resolution segmentation.
• Train from scratch on BraTS data with data augmentation and Dice-based loss, plus optional test-time augmentation (TTA).
• Compare against state-of-the-art 3D segmentation methods on BraTS 2019 and 2020 datasets.

Experimental results

Research questions

• RQ1 Can a Transformer-based global context model be effectively integrated into a 3D CNN framework for volumetric brain tumor segmentation?
• RQ2 Does jointly modeling local 3D context and global dependencies improve Dice scores and Hausdorff distances on BraTS datasets?
• RQ3 What are the impacts of Transformer depth, embedding dimension, token sequence length, and skip-connection placement on segmentation performance?

Key findings

• TransBTS achieves competitive or superior Dice scores and lower Hausdorff distances compared with several 3D baselines on BraTS 2019 validation (e.g., ET 78.93, WT 90.00, TC 81.94 with TTA).
• On BraTS 2019 validation, TransBTS w/ TTA outperforms w/o TTA in Dice for ET, WT, TC and shows notable HD reductions (e.g., ET HD 3.736 mm for w/ TTA).
• On BraTS 2020 validation, TransBTS with/without TTA attain Dice scores around 78.5–78.7 for ET and TC, and around 89–90 for WT, with HD improvements over several baselines (e.g., ET HD ~16.7–17.9 mm).
• A lightweight variant of TransBTS (fewer Transformer layers and smaller FFN) still maintains strong Dice scores while reducing parameters and FLOPs by about 54% and 38%, respectively.
• Ablation studies show longer token sequences (OS=8) and skip-connections from U-Net-style blocks significantly improve performance over other configurations.