[Paper Review] CoMoGAN: continuous model-guided image-to-image translation

Fabio Pizzati, Pietro Cerri+1 more|arXiv (Cornell University)|Mar 11, 2021

Generative Adversarial Networks and Image Synthesis68 references4 citations

TL;DR

CoMoGAN introduces a novel, unsupervised image-to-image translation framework that learns continuous, non-linear domain translations by disentangling content from manifold position using a Functional Instance Normalization (FIN) layer and a Disentanglement Residual Block. It leverages simple physics-inspired models for guidance while allowing discovery of private target features, outperforming prior methods on cyclic and linear translation tasks with state-of-the-art FID and LPIPS scores.

ABSTRACT

CoMoGAN is a continuous GAN relying on the unsupervised reorganization of the target data on a functional manifold. To that matter, we introduce a new Functional Instance Normalization layer and residual mechanism, which together disentangle image content from position on target manifold. We rely on naive physics-inspired models to guide the training while allowing private model/translations features. CoMoGAN can be used with any GAN backbone and allows new types of image translation, such as cyclic image translation like timelapse generation, or detached linear translation. On all datasets, it outperforms the literature. Our code is available at http://github.com/cv-rits/CoMoGAN .

Motivation & Objective

• To enable unsupervised, continuous image-to-image translation without intermediate domain supervision.
• To disentangle image content from position on a functional manifold, enabling control over continuous transformations like timelapse or weather changes.
• To develop a model-agnostic framework that discovers target manifold structure without labeled intermediate points.
• To improve generalization and diversity in image translation by learning to depart from model guidance rather than mimic it.

Proposed method

• Introduces a Functional Instance Normalization (FIN) layer that enables continuous reshaping of the target data manifold by learning a functional mapping of latent codes.
• Employs a Disentanglement Residual Block (DRB) that separates content from manifold position (ϕ) through a novel normalization and residual mechanism.
• Uses a pair-wise ϕ regression network (ϕ-Net) to enforce manifold consistency by estimating distances between generated and model-predicted outputs.
• Optimizes two key losses: model reconstruction (ℒM) to align with physics-inspired models and manifold consistency (ℒϕ) to preserve structural relationships on the manifold.
• Enables end-to-end training with any GAN backbone, supporting both single- and multi-modal translation setups.
• Supports φ-agnostic inference by estimating ϕ from input using ϕ-Net, enabling absolute and relative translations without fixed source conditions.

Experimental results

Research questions

• RQ1 Can continuous, non-linear image translation be learned without intermediate domain supervision or labeled intermediate points?
• RQ2 How can image content and manifold position (ϕ) be disentangled in a way that enables control over continuous transformations like timelapse or weather changes?
• RQ3 Can a simple physics-inspired model guide the learning of complex, private target features without forcing mimicry?
• RQ4 To what extent can a GAN discover and reorganize a functional manifold in an unsupervised manner from mixed source and target data?
• RQ5 Can the framework generalize to truly unsupervised settings where source and target domains are not pre-split?

Key findings

• CoMoGAN achieves state-of-the-art performance on both cyclic (e.g., timelapse) and linear (e.g., depth of field, fog) image translation tasks, outperforming prior SOTA methods.
• On the Day→Timelapse task, CoMoGAN achieves an LPIPS of 0.236 with both ℒM and ℒϕ losses, significantly outperforming ablations (0.020 without ℒM, 0.044 without ℒϕ).
• The model-guided approach with disentanglement allows CoMoGAN to learn exclusive target features, resulting in lower FID (1.41) and better diversity than mimicking models, which diverge in FID when increasing model weight.
• In the absence of dusk/dawn data, training fails drastically, highlighting sensitivity to data sparsity, especially for complex manifold structures.
• φ-agnostic inference enables absolute and relative translations (e.g., 'anytime → day' or '+5°') even on challenging inputs like night images, demonstrating robustness and generalization.
• The framework successfully learns continuous manifolds in truly unsupervised settings (e.g., MNIST-M with domain confusion), generating valid translations without fixed source/target splits.