[Paper Review] MedNeXt: Transformer-driven Scaling of ConvNets for Medical Image Segmentation

Saikat Roy, Gregor Koehler+6 more|arXiv (Cornell University)|Mar 17, 2023

Radiomics and Machine Learning in Medical Imaging17 citations

TL;DR

MedNeXt proposes a fully ConvNeXt 3D encoder-decoder architecture with residual inverted bottlenecks and UpKern kernel upsampling, achieving state-of-the-art medical image segmentation across CT and MRI datasets.

ABSTRACT

There has been exploding interest in embracing Transformer-based architectures for medical image segmentation. However, the lack of large-scale annotated medical datasets make achieving performances equivalent to those in natural images challenging. Convolutional networks, in contrast, have higher inductive biases and consequently, are easily trainable to high performance. Recently, the ConvNeXt architecture attempted to modernize the standard ConvNet by mirroring Transformer blocks. In this work, we improve upon this to design a modernized and scalable convolutional architecture customized to challenges of data-scarce medical settings. We introduce MedNeXt, a Transformer-inspired large kernel segmentation network which introduces - 1) A fully ConvNeXt 3D Encoder-Decoder Network for medical image segmentation, 2) Residual ConvNeXt up and downsampling blocks to preserve semantic richness across scales, 3) A novel technique to iteratively increase kernel sizes by upsampling small kernel networks, to prevent performance saturation on limited medical data, 4) Compound scaling at multiple levels (depth, width, kernel size) of MedNeXt. This leads to state-of-the-art performance on 4 tasks on CT and MRI modalities and varying dataset sizes, representing a modernized deep architecture for medical image segmentation. Our code is made publicly available at: https://github.com/MIC-DKFZ/MedNeXt.

Motivation & Objective

• Demonstrate that a fully ConvNeXt 3D encoder–decoder can outperform Transformer-based and large-kernel baselines in medical image segmentation under data-scarce conditions.
• Introduce residual inverted bottlenecks for up/downsampling to preserve semantic richness across scales.
• Develop UpKern, a kernel upsampling initialization technique to mitigate large-kernel training saturation on limited data.
• Apply compound scaling across depth, width, and receptive field to optimize performance across tasks and modalities.

Proposed method

• Use a 4-layer Encoder-Decoder MedNeXt architecture built exclusively from ConvNeXt blocks.
• Implement Residual Inverted Bottlenecks in Up/Downsampling paths to improve gradient flow and semantic preservation.
• Introduce UpKern: trilinear upsampling-based initialization to expand kernel size without retraining from scratch.
• Employ Compound Scaling to jointly scale depth, width, and kernel size across MedNeXt configurations (S, B, M, L).
• Train with 5-fold cross-validation on four datasets and compare with nnUNet and multiple Transformer/large-kernel baselines.

((a)) MedNeXt macro and block architecture

Experimental results

Research questions

• RQ1 Can a fully ConvNeXt 3D segmentation network surpass Transformer-based and large-kernel methods on medical imaging tasks with limited data?
• RQ2 Do residual inverted bottlenecks and UpKern initialization improve performance and training stability for large kernels in medical segmentation?
• RQ3 Does compound scaling across depth, width, and receptive field yield consistent gains across CT and MRI datasets of varying sizes?

Key findings

• MedNeXt variants achieve state-of-the-art performance across four datasets (BTCV, AMOS22, KiTS19, BraTS21) compared to baselines.
• MedNeXt-L with kernel 5×5×5 and UpKern initialization outperforms nnUNet on public test sets (DSC scores; BTCV 88.76, AMOS22 91.77, KiTS19 91.02, BraTS21 88.01).
• Ablations show residual inverted bottlenecks significantly improve performance over standard resampling.
• UpKern initialization enables large-kernel networks (5×5×5) to surpass small-kernel models; training from scratch with large kernels underperforms.
• Compound scaling across depth, width, and kernel size yields further improvements across datasets.
• On 5-fold CV, MedNeXt variants with 3×3×3 or 5×5×5 kernels outperform all baselines across all datasets.

((b)) UpKern Initialization