[Paper Review] A Multi-channel Network with Image Retrieval for Accurate Brain Tissue Segmentation

Yao Sun, Yang Deng+4 more|arXiv (Cornell University)|Aug 1, 2018

Advanced Neural Network Applications7 references3 citations

TL;DR

This paper proposes a multi-channel convolutional neural network that enhances brain tissue segmentation by incorporating the ground truth of the most similar image retrieved via content-based image retrieval (CBIR) as a fourth input channel. By leveraging prior anatomical texture information from similar scans, the method improves segmentation accuracy by approximately 0.01 in Dice score over standard three-channel networks, with minimal architectural modification and strong robustness across T1, T2-FLAIR, and T1-IR MRI modalities.

ABSTRACT

Magnetic Resonance Imaging (MRI) is widely used in the pathological and functional studies of the brain, such as epilepsy, tumor diagnosis, etc. Automated accurate brain tissue segmentation like cerebro-spinal fluid (CSF), gray matter (GM), white matter (WM) is the basis of these studies and many researchers are seeking it to the best. Based on the truth that multi-channel segmentation network with its own ground truth achieves up to average dice ratio 0.98, we propose a novel method that we add a fourth channel with the ground truth of the most similar image's obtained by CBIR from the database. The results show that the method improves the segmentation performance, as measured by average dice ratio, by approximately 0.01 in the MRBrainS18 database. In addition, our method is concise and robust, which can be used to any network architecture that needs not be modified a lot.

Motivation & Objective

• To improve the accuracy of automated brain tissue segmentation in MRI scans, particularly for CSF, GM, and WM.
• To address the challenge of limited prior anatomical texture information in standard deep learning models for brain MRI segmentation.
• To develop a method that integrates external, relevant anatomical knowledge from similar scans without modifying existing network architectures.
• To evaluate the effectiveness of using CBIR to retrieve the most similar image's ground truth as a fourth input channel for improved segmentation performance.
• To demonstrate the method's robustness and generalizability across different MRI modalities and network architectures.

Proposed method

• The method employs a U-Net-based multi-channel segmentation network with four input channels: T1, T2-FLAIR, T1-IR, and a fourth channel containing the ground truth of the most similar image retrieved via CBIR.
• Content-Based Image Retrieval (CBIR) is used to match each test MRI slice with the most similar image in the database based on image features and post-registration similarity.
• Image similarity is computed using the L1-norm-based formula: $ S(A,B) = 1 - \frac{\|A-B\|_1}{\|A+B\|_1} $, where similarity > threshold triggers the use of the retrieved image's ground truth as the fourth channel.
• The network uses batch normalization and ReLU activation after each convolutional layer, with a mean squared error (MSE) loss function for training.
• When the similarity between the query and retrieved image falls below a threshold, the network defaults to a standard three-channel input.
• Feature maps from contracting and expanding paths are concatenated at the same resolution to preserve both coarse and fine-grained spatial information.

Experimental results

Research questions

• RQ1 Can incorporating the ground truth of a similar image from a database improve segmentation accuracy in brain MRI without modifying the network architecture?
• RQ2 How does the performance of a multi-channel network with CBIR-based fourth-channel input compare to a standard three-channel network on the MRBrainS18 dataset?
• RQ3 To what extent does the use of prior anatomical texture information from similar scans reduce noise and improve boundary smoothness in tissue segmentation?
• RQ4 Does the method remain robust when the most similar image is not perfectly matched, and how does this affect misclassification rates?
• RQ5 Can the proposed method be generalized to other deep learning architectures with minimal architectural changes?

Key findings

• The proposed method improved the average Dice score by approximately 0.01 compared to the standard three-channel network, achieving 0.8440 ± 0.0020 for CSF, 0.8404 ± 0.0023 for GM, and 0.8834 ± 0.0025 for WM.
• The baseline experiment using the test image’s own ground truth as the fourth channel achieved a Dice score of 0.9874 for CSF, 0.9747 for GM, and 0.9626 for WM, demonstrating the strong potential of fourth-channel priors.
• Visualizations showed that the fourth channel significantly reduced random noise and improved boundary smoothness between tissues, especially in GM and WM regions.
• Misclassifications, such as skull regions being segmented as CSF, were observed and attributed to mismatched texture information from non-optimal image matches.
• The method is robust and can be applied to any existing network architecture with minimal modification, as it only requires an additional input channel.
• The results suggest that using the most similar image’s ground truth—rather than a less similar one—would further improve performance, which is a key direction for future work.