[Paper Review] Towards semi-supervised segmentation via image-to-image translation

Eugene Vorontsov, Pavlo Molchanov+7 more|arXiv (Cornell University)|Apr 2, 2019

Domain Adaptation and Few-Shot Learning32 references4 citations

TL;DR

This paper proposes a semi-supervised semantic segmentation framework that leverages image-to-image translation between weak labels (e.g., presence/absence of disease) and segmentation masks to improve performance with limited pixel-level annotations. By sharing encoders and decoders across translation and segmentation tasks and selectively modeling domain-specific variations, the method achieves significant gains on brain tumor segmentation in MRI, outperforming standard autoencoding-based semi-supervised approaches.

ABSTRACT

In many cases, especially with medical images, it is prohibitively challenging to produce a sufficiently large training sample of pixel-level annotations to train deep neural networks for semantic image segmentation. On the other hand, some information is often known about the contents of images. We leverage information on whether an image presents the segmentation target or whether it is absent from the image to improve segmentation performance by augmenting the amount of data usable for model training. Specifically, we propose a semi-supervised framework that employs image-to-image translation between weak labels (e.g., presence vs. absence of cancer), in addition to fully supervised segmentation on some examples. We conjecture that this translation objective is well aligned with the segmentation objective as both require the same disentangling of image variations. Building on prior image-to-image translation work, we re-use the encoder and decoders for translating in either direction between two domains, employing a strategy of selectively decoding domain-specific variations. For presence vs. absence domains, the encoder produces variations that are common to both and those unique to the presence domain. Furthermore, we successfully re-use one of the decoders used in translation for segmentation. We validate the proposed method on synthetic tasks of varying difficulty as well as on the real task of brain tumor segmentation in magnetic resonance images, where we show significant improvements over standard semi-supervised training with autoencoding.

Motivation & Objective

• To address the challenge of limited pixel-level annotations in medical image segmentation, especially in MRI.
• To improve segmentation performance by leveraging weak supervision signals such as presence or absence of a target (e.g., tumor).
• To unify image-to-image translation and semantic segmentation through shared encoder-decoder architectures.
• To explore whether translation between weak labels and segmentation masks can serve as a meaningful self-supervision signal.

Proposed method

• The framework uses a shared encoder to extract features common to both presence and absence domains, along with domain-specific features for the presence class.
• It employs two decoders: one for translating from presence/absence labels to images, and another for translating from images to segmentation masks.
• The model trains a cycle-consistent GAN-based image-to-image translation network between weak labels and segmentation masks to enforce structural consistency.
• Domain-specific variations are selectively decoded using attention mechanisms or skip connections to preserve target-specific details.
• The segmentation decoder is re-used from the translation branch, enabling parameter sharing and joint optimization.
• The method integrates both translation and segmentation objectives in a unified training objective, improving feature disentanglement and generalization.

Experimental results

Research questions

• RQ1 Can image-to-image translation between weak labels (presence/absence) and segmentation masks improve semi-supervised segmentation performance?
• RQ2 How does the shared representation learning between translation and segmentation tasks enhance feature disentanglement?
• RQ3 Does the proposed method outperform standard semi-supervised training with autoencoding on medical image segmentation?
• RQ4 To what extent does the model benefit from the alignment between translation and segmentation objectives in terms of feature learning?
• RQ5 Can the framework generalize across synthetic and real-world medical imaging tasks?

Key findings

• The proposed method achieves significant performance gains on brain tumor segmentation in MRI compared to standard semi-supervised training with autoencoding.
• The model improves segmentation accuracy even with only a small number of fully annotated training samples, demonstrating strong data efficiency.
• The joint training of translation and segmentation tasks leads to better disentanglement of content and domain-specific features.
• The re-use of the segmentation decoder from the translation branch reduces parameter count and improves training stability.
• The method generalizes well to synthetic tasks of varying difficulty, confirming its robustness and scalability.
• The ablation study confirms that the cycle-consistent translation objective contributes meaningfully to performance improvement.