[Paper Review] HUMUS-Net: Hybrid unrolled multi-scale network architecture for accelerated MRI reconstruction

Zalan Fabian, Tinaz, Berk+1 more|arXiv (Cornell University)|Mar 15, 2022

Advanced X-ray and CT Imaging45 citations

TL;DR

HUMUS-Net combines convolutional and Transformer-based modules in an unrolled, multi-scale architecture to improve accelerated MRI reconstruction, achieving state-of-the-art results on fastMRI knee data.

ABSTRACT

In accelerated MRI reconstruction, the anatomy of a patient is recovered from a set of under-sampled and noisy measurements. Deep learning approaches have been proven to be successful in solving this ill-posed inverse problem and are capable of producing very high quality reconstructions. However, current architectures heavily rely on convolutions, that are content-independent and have difficulties modeling long-range dependencies in images. Recently, Transformers, the workhorse of contemporary natural language processing, have emerged as powerful building blocks for a multitude of vision tasks. These models split input images into non-overlapping patches, embed the patches into lower-dimensional tokens and utilize a self-attention mechanism that does not suffer from the aforementioned weaknesses of convolutional architectures. However, Transformers incur extremely high compute and memory cost when 1) the input image resolution is high and 2) when the image needs to be split into a large number of patches to preserve fine detail information, both of which are typical in low-level vision problems such as MRI reconstruction, having a compounding effect. To tackle these challenges, we propose HUMUS-Net, a hybrid architecture that combines the beneficial implicit bias and efficiency of convolutions with the power of Transformer blocks in an unrolled and multi-scale network. HUMUS-Net extracts high-resolution features via convolutional blocks and refines low-resolution features via a novel Transformer-based multi-scale feature extractor. Features from both levels are then synthesized into a high-resolution output reconstruction. Our network establishes new state of the art on the largest publicly available MRI dataset, the fastMRI dataset. We further demonstrate the performance of HUMUS-Net on two other popular MRI datasets and perform fine-grained ablation studies to validate our design.

Motivation & Objective

• Motivate improved reconstruction for accelerated MRI by addressing limitations of purely convolutional models in modeling long-range dependencies.
• Propose a hybrid architecture that preserves high-resolution details while leveraging multi-scale Transformer-based feature extraction.
• Introduce an unrolled reconstruction framework with an image-domain denoiser that fuses high- and low-resolution features.
• Demonstrate state-of-the-art performance on large-scale MRI datasets and provide ablation studies validating design choices.

Proposed method

• Introduce HUMUS-Net, a cascade of unrolled iterations for k-space MRI reconstruction.
• HUMUS-Block extracts high-resolution features via a 3x3 convolutional layer and low-resolution features via a convolutional downsampling path.
• A novel multi-scale hybrid feature extractor MUST performs encoder-decoder processing in both image and token representations using Swin-transformer blocks with skip connections.
• Low-resolution features are processed in a Transformer-convolutional hybrid setup to capture long-range dependencies while keeping computations feasible.
• Reconstruction synthesizes high-resolution and deep low-resolution features with a residual image learning module: x_out = x_in + f_R(F_H, F_D).
• The forward model uses data consistency and coil sensitivity expansion/reduction, with optional adjacent-slice reconstruction to improve context.

Experimental results

Research questions

• RQ1 Can a hybrid convolutional-Transformer architecture improve reconstruction quality for accelerated MRI compared to purely CNN-based or Transformer-based models?
• RQ2 Does multi-scale processing and unrolled k-space optimization yield state-of-the-art results on large public MRI datasets like fastMRI?
• RQ3 What is the impact of processing high-resolution image features together with low-resolution Transformer-based features on reconstruction fidelity?
• RQ4 Does adjacent slice reconstruction (ASR) provide consistent gains across datasets?
• RQ5 How do ablations (single-scale SwinIR vs MUST vs HUMUS-Block) inform the importance of multi-scale, hybrid processing?

Key findings

• HUMUS-Net achieves SSIM=0.8936, PSNR=37.0, NMSE=0.0086 on fastMRI knee test data, outperforming several baselines.
• A variant HUMUS-Net-L attains SSIM=0.8951, PSNR=37.4, NMSE=0.0080 with larger embedding dimension.
• On fastMRI validation, HUMUS-Net reaches SSIM=0.8934 (vs E2E-VarNet 0.8908).
• Ablations show multi-scale hybrid processing (MUST) outperforms single-scale SwinIR and unrolled variants, with HUMUS-Net achieving SSIM=0.9449±0.0064 on Stanford 3D MRI.
• ASR further improves reconstruction by leveraging context from adjacent slices without changing token count.
• Overall HUMUS-Net achieves new state-of-the-art SSIM on fastMRI knee and demonstrates robustness across Stanford 2D/3D datasets.