[Paper Review] Geometric Visual Similarity Learning in 3D Medical Image Self-supervised Pre-training

Yuting He, Guanyu Yang+5 more|arXiv (Cornell University)|Mar 2, 2023

AI in cancer detection4 citations

TL;DR

This paper proposes Geometric Visual Similarity Learning (GVSL), a novel self-supervised pre-training framework for 3D medical images that leverages topological invariance to improve inter-image similarity measurement. By introducing a Z-matching head that jointly learns global and local geometric correspondence, GVSL enables consistent representation learning across semantic regions, significantly boosting performance in inner-scene, inter-scene, and global-local transfer tasks on four 3D medical imaging benchmarks.

ABSTRACT

Learning inter-image similarity is crucial for 3D medical images self-supervised pre-training, due to their sharing of numerous same semantic regions. However, the lack of the semantic prior in metrics and the semantic-independent variation in 3D medical images make it challenging to get a reliable measurement for the inter-image similarity, hindering the learning of consistent representation for same semantics. We investigate the challenging problem of this task, i.e., learning a consistent representation between images for a clustering effect of same semantic features. We propose a novel visual similarity learning paradigm, Geometric Visual Similarity Learning, which embeds the prior of topological invariance into the measurement of the inter-image similarity for consistent representation of semantic regions. To drive this paradigm, we further construct a novel geometric matching head, the Z-matching head, to collaboratively learn the global and local similarity of semantic regions, guiding the efficient representation learning for different scale-level inter-image semantic features. Our experiments demonstrate that the pre-training with our learning of inter-image similarity yields more powerful inner-scene, inter-scene, and global-local transferring ability on four challenging 3D medical image tasks. Our codes and pre-trained models will be publicly available on https://github.com/YutingHe-list/GVSL.

Motivation & Objective

• Address the challenge of unreliable inter-image similarity measurement in 3D medical image self-supervised pre-training due to semantic-independent appearance variations.
• Overcome limitations of existing methods—clustering-based, contrastive, and generation-based—by embedding topological invariance as a prior for robust similarity learning.
• Develop a unified framework that jointly optimizes geometric matching and self-restoration to enhance feature consistency for same semantic regions across different images.
• Improve transferability of pre-trained models across diverse downstream tasks, including inner-scene, inter-scene, and global-local transfer settings.

Proposed method

• Propose a new paradigm, Geometric Visual Similarity Learning (GVSL), that incorporates topological invariance as an inductive bias to stabilize inter-image similarity measurement in 3D medical images.
• Design a Z-matching head that learns both global and local geometric correspondence between image pairs via a differentiable geometric matching mechanism.
• Integrate a self-restoration pretext task to provide a basic, robust representation of visual semantics, which supports the optimization of the geometric matching task.
• Use normalized cross-correlation (NCC) loss as the supervision signal for geometric matching, enabling alignment of semantic regions across images with appearance variations.
• Train the model end-to-end using a multi-task objective combining self-restoration and geometric matching, with the Z-matching head guiding consistent feature learning.
• Leverage the spatial completeness and anatomical consistency of 3D medical images to exploit topological invariance for reliable semantic correspondence discovery.

Experimental results

Research questions

• RQ1 How can topological invariance in 3D medical images be leveraged to improve the reliability of inter-image similarity measurement in self-supervised pre-training?
• RQ2 Can a geometric matching head that jointly learns global and local correspondence enhance the consistency of representations for the same semantic regions across different images?
• RQ3 To what extent does combining geometric matching with a self-restoration pretext task improve clustering of semantic features and downstream transfer performance?
• RQ4 How does GVSL compare to existing self-supervised methods in terms of inner-scene, inter-scene, and global-local transferability on 3D medical imaging tasks?

Key findings

• GVSL significantly improves the clustering effect of local features, with clearly separated clusters for different semantic regions, especially when both self-restoration and Z-matching are used.
• The Z-matching head enables effective learning of geometric correspondence even under large appearance variations, as shown by converging NCC loss and improved feature alignment.
• The combination of self-restoration and geometric matching leads to more representative and distinguishing features than either task alone.
• GVSL achieves strong performance across four challenging 3D medical image tasks, demonstrating superior inner-scene, inter-scene, and global-local transferability.
• The ablation study confirms that the fundamental self-restoration task is essential for stabilizing the optimization of the geometric matching task.
• The method shows robustness to scanning protocol variations and maintains consistent representation for same semantic regions despite appearance differences.