[Paper Review] Pretraining is All You Need for Image-to-Image Translation

Tengfei Wang, Ting Zhang+5 more|arXiv (Cornell University)|May 25, 2022

Generative Adversarial Networks and Image SynthesisComputer Science90 citations

TL;DR

The paper shows that a pretrained diffusion model can serve as a universal generative prior for various image-to-image translation tasks, with task-specific adapters and training tweaks yielding state-of-the-art results across multiple benchmarks.

ABSTRACT

We propose to use pretraining to boost general image-to-image translation. Prior image-to-image translation methods usually need dedicated architectural design and train individual translation models from scratch, struggling for high-quality generation of complex scenes, especially when paired training data are not abundant. In this paper, we regard each image-to-image translation problem as a downstream task and introduce a simple and generic framework that adapts a pretrained diffusion model to accommodate various kinds of image-to-image translation. We also propose adversarial training to enhance the texture synthesis in the diffusion model training, in conjunction with normalized guidance sampling to improve the generation quality. We present extensive empirical comparison across various tasks on challenging benchmarks such as ADE20K, COCO-Stuff, and DIODE, showing the proposed pretraining-based image-to-image translation (PITI) is capable of synthesizing images of unprecedented realism and faithfulness.

Motivation & Objective

• Motivate and demonstrate that large-scale pretraining can boost conditional image synthesis across diverse tasks.
• Leverage a pretrained diffusion model (GLIDE) as a semantic latent prior for downstream translation tasks.
• Develop a practical two-stage finetuning protocol to map input conditions into the pretrained latent space.
• Enhance generation quality with an adversarial diffusion upsampler and perceptual/adversarial losses.
• Propose normalized classifier-free guidance to improve conditioning fidelity and sample quality.

Proposed method

• Use a generative pretrained diffusion prior (GLIDE) trained on diverse text-image pairs as the decoder.
• Attach a task-specific encoder head to map inputs (e.g., segmentation masks, sketches, depth maps) into the pretrained latent space.
• Two-stage finetuning: (1) train encoder while keeping decoder fixed; (2) fine-tune encoder and decoder end-to-end.
• Adversarial diffusion upsampler during high-resolution generation with degradations and perceptual/adversarial losses to improve textures.
• Introduce normalized classifier-free guidance to correct mean/variance shifts during conditional sampling and allow stronger guidance without saturation.
• Employ a hierarchical generation setup with diffusion-based upsampling to produce 256x256 images from 64x64 base outputs.

Experimental results

Research questions

• RQ1 Can a single pretrained diffusion model serve as a universal prior for multiple image-to-image translation tasks?
• RQ2 What training strategies (two-stage fine-tuning, adversarial upsampling, perceptual/adversarial losses) maximize translation quality?
• RQ3 How does PITI compare to task-specific or scratch-trained baselines on standard benchmarks (ADE20K, COCO-Stuff, DIODE)?
• RQ4 Does pretraining enable high-quality results with limited downstream data (few-shot scenarios)?

Key findings

• PITI outperforms baselines trained from scratch by a large margin across tasks and datasets.
• Two-stage finetuning yields the best results, enabling semantic alignment while preserving the pretrained prior.
• Adversarial diffusion upsampling and perceptual/adversarial losses significantly improve texture and detail in high-resolution outputs.
• Normalized classifier-free guidance improves sample quality under strong guidance without introducing harmful mean/variance shifts.
• On ADE20K, COCO-Stuff, and DIODE, PITI achieves state-of-the-art-like FID improvements over both task-specific and scratch baselines; results shown in Table 1 indicate substantial gains.