[Paper Review] Breaking the Dilemma of Medical Image-to-image Translation

Lingke Kong, Chenyu Lian+4 more|arXiv (Cornell University)|Oct 13, 2021

Radiomics and Machine Learning in Medical ImagingMedicine24 references74 citations

TL;DR

RegGAN introduces a loss-correction framework with a registration network to handle misaligned targets in medical image translation, outperforming Pix2Pix and Cycle-consistency across aligned, misaligned, and unpaired data.

ABSTRACT

Supervised Pix2Pix and unsupervised Cycle-consistency are two modes that dominate the field of medical image-to-image translation. However, neither modes are ideal. The Pix2Pix mode has excellent performance. But it requires paired and well pixel-wise aligned images, which may not always be achievable due to respiratory motion or anatomy change between times that paired images are acquired. The Cycle-consistency mode is less stringent with training data and works well on unpaired or misaligned images. But its performance may not be optimal. In order to break the dilemma of the existing modes, we propose a new unsupervised mode called RegGAN for medical image-to-image translation. It is based on the theory of "loss-correction". In RegGAN, the misaligned target images are considered as noisy labels and the generator is trained with an additional registration network to fit the misaligned noise distribution adaptively. The goal is to search for the common optimal solution to both image-to-image translation and registration tasks. We incorporated RegGAN into a few state-of-the-art image-to-image translation methods and demonstrated that RegGAN could be easily combined with these methods to improve their performances. Such as a simple CycleGAN in our mode surpasses latest NICEGAN even though using less network parameters. Based on our results, RegGAN outperformed both Pix2Pix on aligned data and Cycle-consistency on misaligned or unpaired data. RegGAN is insensitive to noises which makes it a better choice for a wide range of scenarios, especially for medical image-to-image translation tasks in which well pixel-wise aligned data are not available

Motivation & Objective

• Motivate the need for an image-to-image translation mode that handles misaligned or unpaired medical images beyond Pix2Pix and Cycle-consistency.
• Propose RegGAN as a loss-correction framework that jointly trains a generator and a registration network to adaptively model deformation noise.
• Demonstrate that RegGAN can be integrated with existing translation models to improve performance with fewer parameters.
• Evaluate RegGAN on a public medical imaging dataset under aligned, misaligned, and unpaired conditions, and across varying noise levels.

Proposed method

• Formulate image-to-image translation as supervised learning with noisy labels when target images are misaligned.
• Introduce a registration network R after the generator G to learn a deformation field that corrects label noise.
• Define a correction loss that minimizes the difference between the misaligned target and the warped generator output, plus a smoothness loss on the deformation field.
• Combine Corr loss, Smooth loss, and adversarial loss into a total loss for joint optimization of G, R, and D.
• Treat deformation noise as the noise model; use loss-correction to align the noisy labels with the true target distribution.

Experimental results

Research questions

• RQ1 Can RegGAN achieve competitive or superior translation quality when target images are misaligned or unpaired, compared to Pix2Pix and Cycle-consistency?
• RQ2 Does incorporating a registration network to model deformation noise improve robustness to misalignment and various noise levels in medical image translation?
• RQ3 Can RegGAN be effectively integrated with existing image-to-image translation architectures to boost performance with fewer parameters?
• RQ4 How does RegGAN perform on aligned data versus misaligned/unpaired data across multiple datasets and methods?

Key findings

• RegGAN consistently improves translation performance when targets are misaligned or unpaired across evaluated methods.
• Incorporating the registration network (+R) yields substantial performance gains across metrics and methods, often surpassing Cycle-consistency baselines.
• The NC+R configuration (non-cycle-consistency plus registration) frequently outperforms the C+R configuration, suggesting that cycle-consistency may hinder performance when combined with RegGAN.
• RegGAN remains robust to increasing noise levels and non-affine noise, maintaining superior metrics over Pix2Pix and CycleGAN baselines.
• On unpaired data, RegGAN outperforms Pix2Pix and CycleGAN, though gains are more modest than in paired settings.
• The approach can be integrated with different architectures (CycleGAN, MUNIT, UNIT, NICEGAN) and achieves better or comparable results with fewer parameters.