[Paper Review] CDVAE: Co-embedding Deep Variational Auto Encoder for Conditional Variational Generation

Jiajun Lu, Aditya Deshpande+1 more|arXiv (Cornell University)|Dec 1, 2016

Generative Adversarial Networks and Image Synthesis33 references3 citations

TL;DR

This paper proposes CDVAE, a co-embedding deep variational autoencoder that prevents code space collapse in conditional variational generation by enforcing metric constraints on the latent code space. By guiding the model to preserve input similarity in the code space and using a Mixture Density Network for multimodal output, CDVAE achieves superior diversity and quality in image relighting and resaturation tasks, outperforming strong baselines in both quantitative and qualitative evaluations.

ABSTRACT

Problems such as predicting a new shading field (Y) for an image (X) are ambiguous: many very distinct solutions are good. Representing this ambiguity requires building a conditional model P(Y|X) of the prediction, conditioned on the image. Such a model is difficult to train, because we do not usually have training data containing many different shadings for the same image. As a result, we need different training examples to share data to produce good models. This presents a danger we call "code space collapse" - the training procedure produces a model that has a very good loss score, but which represents the conditional distribution poorly. We demonstrate an improved method for building conditional models by exploiting a metric constraint on training data that prevents code space collapse. We demonstrate our model on two example tasks using real data: image saturation adjustment, image relighting. We describe quantitative metrics to evaluate ambiguous generation results. Our results quantitatively and qualitatively outperform different strong baselines.

Motivation & Objective

• To address the challenge of learning multimodal conditional distributions in vision tasks where training data is scattered and lacks multiple outputs per input.
• To identify and mitigate 'code space collapse'—a failure mode in conditional VAEs where the model ignores latent variables and produces low-diversity outputs.
• To improve generalization and diversity in conditional generation by enforcing that similar inputs produce similar codes and dissimilar inputs produce dissimilar codes.
• To develop a method that enables high-quality, diverse generation of spatial fields (e.g., shading, saturation) conditioned on input images, even with limited paired training data.

Proposed method

• CDVAE introduces a metric constraint that encourages the code space to preserve input similarity: similar inputs map to similar codes, and dissimilar inputs to dissimilar codes.
• The model uses a deep encoder to produce a code $ c(x) $ from input image $ X $, which is then used as input to a Mixture Density Network (MDN) to model multimodal output $ Y $.
• The MDN component explicitly models the conditional distribution $ P(Y|X) $ by predicting mixture parameters (means, variances, weights) of a Gaussian mixture, enabling diverse and structured outputs.
• A reconstruction loss ensures fidelity to ground-truth outputs, while a KL divergence term regularizes the latent code distribution to match a standard normal distribution.
• The metric constraint is implemented as a contrastive loss that pulls together codes of similar inputs and pushes apart codes of dissimilar inputs.
• The model is trained end-to-end using backpropagation, with the metric constraint preventing the network from collapsing into a single code for similar inputs.

Experimental results

Research questions

• RQ1 Can a conditional variational autoencoder be made robust to scattered training data by preventing code space collapse?
• RQ2 How can a metric constraint on the code space improve the diversity and quality of conditional generation in ambiguous vision tasks?
• RQ3 To what extent does enforcing code similarity for similar inputs enhance generalization and reduce artifacts in image relighting and resaturation?
• RQ4 Does using a Mixture Density Network in combination with a metric-constrained code space lead to better multimodal modeling than standard CVAE or GAN-based approaches?

Key findings

• CDVAE achieves the lowest best error to ground truth across all sample numbers in both photo relighting and image resaturation tasks, with errors of 1.11 and 3.82 (×10⁻²) respectively at 100 samples.
• The variance of predictions across 100 samples is consistently lower for CDVAE (1.77 for relighting, 3.55 for resaturation) compared to CVAE (0.19 and 1.20), indicating better diversity and reduced mode collapse.
• Qualitative results show that CVAE and CGAN suffer from mode collapse and produce low-diversity, artifact-ridden outputs, while CDVAE generates more varied, spatially coherent, and realistic results.
• The ablation study shows that removing the embedding guidance leads to less variety and more artifacts, confirming the importance of the metric constraint in preserving code space structure.
• CDVAE with 12 embedding dimensions (CDVAE12) achieves the best trade-off between error and variance, outperforming both CVAE and baseline GANs in all metrics.
• The model demonstrates robustness to input perturbations and maintains consistent output diversity, unlike models with collapsed codes that produce unstable or identical outputs for small input changes.