[Paper Review] CoMIR: Contrastive Multimodal Image Representation for Registration

Nicolas Pielawski, Elisabeth Wetzer+5 more|arXiv (Cornell University)|Jun 11, 2020

Image Processing Techniques and Applications63 references55 citations

TL;DR

CoMIR learns shared, dense representations for multimodal images via contrastive learning, enabling monomodal registration with rotationally equivariant features.

ABSTRACT

We propose contrastive coding to learn shared, dense image representations, referred to as CoMIRs (Contrastive Multimodal Image Representations). CoMIRs enable the registration of multimodal images where existing registration methods often fail due to a lack of sufficiently similar image structures. CoMIRs reduce the multimodal registration problem to a monomodal one, in which general intensity-based, as well as feature-based, registration algorithms can be applied. The method involves training one neural network per modality on aligned images, using a contrastive loss based on noise-contrastive estimation (InfoNCE). Unlike other contrastive coding methods, used for, e.g., classification, our approach generates image-like representations that contain the information shared between modalities. We introduce a novel, hyperparameter-free modification to InfoNCE, to enforce rotational equivariance of the learnt representations, a property essential to the registration task. We assess the extent of achieved rotational equivariance and the stability of the representations with respect to weight initialization, training set, and hyperparameter settings, on a remote sensing dataset of RGB and near-infrared images. We evaluate the learnt representations through registration of a biomedical dataset of bright-field and second-harmonic generation microscopy images; two modalities with very little apparent correlation. The proposed approach based on CoMIRs significantly outperforms registration of representations created by GAN-based image-to-image translation, as well as a state-of-the-art, application-specific method which takes additional knowledge about the data into account. Code is available at: https://github.com/MIDA-group/CoMIR.

Motivation & Objective

• Motivate multimodal image registration where modalities lack similar structures.
• Propose a contrastive learning framework to produce shared representations (CoMIRs) for different modalities.
• Enable application of standard monomodal registration algorithms to multimodal pairs.
• Introduce a rotation-equivariance constraint that does not require extra hyperparameters or architecture changes.

Proposed method

• Use one neural network per modality to learn CoMIRs from aligned image pairs.
• Formulate a two-modality contrastive loss (InfoNCE) with a critic h(y^1, y^2) to maximize mutual information between CoMIRs.
• Introduce a rotation equivariance constraint by maximizing similarities under 90-degree rotational transforms without extra hyperparameters.
• Sample negative patches from within training images to provide diverse negative examples for the contrastive loss.
• Experiment with critics: MSE and cosine similarity; report that MSE-based CoMIRs favor registration performance.

Experimental results

Research questions

• RQ1 Can aligned multimodal image pairs be mapped to a shared, dense representation that supports monomodal registration?
• RQ2 Does a contrastive loss with InfoNCE produce rotation-equivariant CoMIRs suitable for registration across modalities?
• RQ3 How do different critics (MSE vs cosine) and hyperparameters affect CoMIR quality and registration performance?
• RQ4 What is the stability and reproducibility of CoMIRs with respect to initialization and training data?] ,

Key findings

• CoMIRs enable successful registration of multimodal image pairs by reducing multimodal registration to monomodal methods.
• A rotation-equivariance constraint can be incorporated into the contrastive loss in a hyperparameter-free way, yielding equivariant representations beyond multiples of 90 degrees.
• MSE-based CoMIRs generally provide better registration performance than cosine-based variants in the evaluated setups.
• On biomedical BF and SHG data, CoMIR-based registration significantly outperforms GAN-based image translation approaches and a state-of-the-art data-specific method.
• CoMIR training is stable with respect to hyperparameters and weight initialization, and can work with very small training sets (even a single image pair).
• The approach demonstrates robustness across datasets (Zurich aerial RGB/NIR and biomedical BF/SHG) and supports both intensity- and feature-based monomodal registrations.