[Paper Review] Semi-supervised Learning for Few-shot Image-to-Image Translation

Yaxing Wang, Salman Khan+3 more|arXiv (Cornell University)|Mar 30, 2020

Generative Adversarial Networks and Image Synthesis43 references4 citations

TL;DR

This paper proposes SEMIT, a semi-supervised few-shot image-to-image translation method that reduces reliance on labeled source images by leveraging unlabeled data through noise-tolerant pseudo-labeling and cycle consistency. It achieves performance comparable to fully supervised models using only 20% of the labeled source data, and matches state-of-the-art results with just 10% labeled data across four datasets.

ABSTRACT

In the last few years, unpaired image-to-image translation has witnessed remarkable progress. Although the latest methods are able to generate realistic images, they crucially rely on a large number of labeled images. Recently, some methods have tackled the challenging setting of few-shot image-to-image translation, reducing the labeled data requirements for the target domain during inference. In this work, we go one step further and reduce the amount of required labeled data also from the source domain during training. To do so, we propose applying semi-supervised learning via a noise-tolerant pseudo-labeling procedure. We also apply a cycle consistency constraint to further exploit the information from unlabeled images, either from the same dataset or external. Additionally, we propose several structural modifications to facilitate the image translation task under these circumstances. Our semi-supervised method for few-shot image translation, called SEMIT, achieves excellent results on four different datasets using as little as 10% of the source labels, and matches the performance of the main fully-supervised competitor using only 20% labeled data. Our code and models are made public at: https://github.com/yaxingwang/SEMIT.

Motivation & Objective

• Address the high cost and impracticality of acquiring large-scale labeled datasets for image-to-image translation.
• Overcome the limitation of existing few-shot methods that require abundant labeled source data during training.
• Explore a novel semi-supervised few-shot I2I translation setting where only limited labeled data is available from the source domain during training.
• Improve generalization and performance by effectively utilizing unlabeled source images, including external data, through robust pseudo-labeling and cycle consistency.
• Develop generic architectural enhancements—such as OctConv and entropy regulation—to improve learning efficiency under low-data regimes.

Proposed method

• Apply a noise-tolerant pseudo-labeling procedure to assign soft labels to unlabeled source images based on an initial small set of labeled images.
• Integrate cycle consistency loss to enforce content preservation and exploit information from unlabeled images, both from the same dataset and external sources.
• Incorporate Octave Convolution (OctConv) to disentangle high- and low-frequency features, focusing learning on high-frequency image details critical for translation.
• Apply entropy regulation to encourage domain-invariant latent representations, improving generalization to unseen target domains.
• Use a multi-stage training strategy combining pseudo-labeling, cycle consistency, and classification loss (for labeled data) to jointly optimize the translation model.
• Train the model on both in-domain and external unlabeled data, without applying classification loss on external data to avoid distribution shift.

Experimental results

Research questions

• RQ1 Can semi-supervised learning reduce the dependency on labeled source images in few-shot image-to-image translation without sacrificing performance?
• RQ2 How effective is noise-tolerant pseudo-labeling in generating reliable supervision signals from unlabeled source images in a low-data regime?
• RQ3 To what extent can cycle consistency loss improve performance when applied to unlabeled data in few-shot I2I translation?
• RQ4 Can architectural modifications like OctConv and entropy regulation enhance model generalization under limited labeled data?
• RQ5 Can the model generalize to unseen target domains when trained on a combination of in-domain and external unlabeled data?

Key findings

• SEMIT achieves performance comparable to fully supervised FUNIT using only 20% of the labeled source data, with an mFID of 65.21 on the Animals dataset versus FUNIT’s 66.14 at 100%.
• With just 10% labeled source data, SEMIT achieves an mFID of 71.49 on the Flowers dataset, outperforming FUNIT (mFID 72.64) at 100% labeled data.
• On the Birds dataset, SEMIT with 20% labeled data (mFID 65.42) matches FUNIT’s performance at 100% labeled data (mFID 66.14).
• SEMIT generates more realistic and sharp target-specific images than FUNIT under low-data regimes, especially on challenging datasets like Flowers.
• Joint training on expanded datasets (e.g., Animals++ and Birds++) improves visual quality, demonstrating the model’s ability to leverage external unlabeled data.
• The method maintains strong performance even in 1-shot and 5-shot settings, with SEMIT-1 achieving competitive results across all metrics on Animals and Birds.