[Paper Review] A clinically motivated self-supervised approach for content-based image retrieval of CT liver images

Kristoffer Wickstrøm, Eirik Agnalt Østmo+4 more|arXiv (Cornell University)|Jul 11, 2022

Radiomics and Machine Learning in Medical Imaging4 citations

TL;DR

This paper proposes a clinically motivated self-supervised learning framework for content-based image retrieval (CBIR) in CT liver images that improves performance and generalization by incorporating domain knowledge through Hounsfield unit clipping to focus on liver tissue. It introduces the first representation learning explainability analysis in CBIR, revealing clinically relevant feature extraction and enhanced trustworthiness.

ABSTRACT

Deep learning-based approaches for content-based image retrieval (CBIR) of CT liver images is an active field of research, but suffers from some critical limitations. First, they are heavily reliant on labeled data, which can be challenging and costly to acquire. Second, they lack transparency and explainability, which limits the trustworthiness of deep CBIR systems. We address these limitations by (1) proposing a self-supervised learning framework that incorporates domain-knowledge into the training procedure and (2) providing the first representation learning explainability analysis in the context of CBIR of CT liver images. Results demonstrate improved performance compared to the standard self-supervised approach across several metrics, as well as improved generalisation across datasets. Further, we conduct the first representation learning explainability analysis in the context of CBIR, which reveals new insights into the feature extraction process. Lastly, we perform a case study with cross-examination CBIR that demonstrates the usability of our proposed framework. We believe that our proposed framework could play a vital role in creating trustworthy deep CBIR systems that can successfully take advantage of unlabeled data.

Motivation & Objective

• Address the critical limitation of deep CBIR systems requiring large amounts of labeled data for training.
• Improve model performance and generalization across datasets by integrating domain-specific knowledge into self-supervised learning.
• Provide the first representation learning explainability analysis in the context of CBIR for CT liver images to enhance transparency and clinical trust.
• Demonstrate the usability of the framework in a real-world clinical scenario involving cross-examination of patient scans.
• Enable effective use of unlabeled data in CBIR by designing a method that focuses on clinically relevant anatomical structures.

Proposed method

• Propose a self-supervised learning framework that applies Hounsfield unit (HU) clipping to isolate liver tissue from CT images during training, reducing noise from non-liver structures.
• Use contrastive learning with augmented views of liver-relevant regions as positive pairs to train a feature extractor that learns discriminative, organ-specific representations.
• Incorporate clinical knowledge by designing data augmentation and clipping strategies tailored to the HU range of liver tissue (typically 40–120 HU), enhancing feature relevance.
• Apply the RELAX framework for representation learning explainability to interpret which input features influence the learned representations.
• Train the model end-to-end using contrastive loss to maximize similarity between positive pairs while minimizing similarity with negative pairs.
• Evaluate the framework on multiple datasets, including cross-dataset generalization, to assess robustness and clinical applicability.

Experimental results

Research questions

• RQ1 Can a self-supervised learning framework that leverages domain-specific knowledge (e.g., HU range of liver tissue) improve CBIR performance on CT liver images compared to standard self-supervised baselines?
• RQ2 How does the proposed method generalize across different datasets and scanning protocols in clinical settings?
• RQ3 To what extent can representation learning explainability techniques reveal clinically meaningful features in self-supervised CBIR models?
• RQ4 Can the framework successfully retrieve diagnostically relevant images in a real clinical cross-examination scenario, such as tracking liver metastasis over time?
• RQ5 Does focusing on liver-specific regions via HU clipping lead to more interpretable and trustworthy representations in CBIR?

Key findings

• The proposed self-supervised framework achieves improved retrieval performance across multiple metrics compared to standard self-supervised baselines, including higher mAP and recall rates.
• The model demonstrates superior generalization ability when evaluated on external datasets, indicating robustness to variations in scanning protocols and imaging settings.
• The representation learning explainability analysis using RELAX reveals that the model learns features aligned with clinically relevant anatomical and pathological patterns in the liver.
• The CBIR system successfully retrieves images matching physician-selected references in a cross-examination case study, confirming clinical usability.
• The system retrieves similar images to a radiologist’s selection, though it lacks spatial coherence in ordering, suggesting potential for improvement via modeling local context.
• The integration of HU-based clipping significantly enhances feature quality by suppressing irrelevant anatomical structures, leading to more focused and meaningful representations.