[Paper Review] CoDi: Co-evolving Contrastive Diffusion Models for Mixed-type Tabular Synthesis

Chaejeong Lee, Jayoung Kim+1 more|arXiv (Cornell University)|Apr 25, 2023

Topic Modeling4 citations

TL;DR

CoDi proposes a novel framework for mixed-type tabular data synthesis using two co-evolving diffusion models—one for continuous and one for discrete variables—conditioned on each other and jointly optimized via contrastive learning with negative sampling. The method achieves state-of-the-art performance in sampling quality and diversity across 11 real-world datasets, outperforming prior methods including STaSy while reducing training difficulty and improving sampling speed by ~90%.

ABSTRACT

With growing attention to tabular data these days, the attempt to apply a synthetic table to various tasks has been expanded toward various scenarios. Owing to the recent advances in generative modeling, fake data generated by tabular data synthesis models become sophisticated and realistic. However, there still exists a difficulty in modeling discrete variables (columns) of tabular data. In this work, we propose to process continuous and discrete variables separately (but being conditioned on each other) by two diffusion models. The two diffusion models are co-evolved during training by reading conditions from each other. In order to further bind the diffusion models, moreover, we introduce a contrastive learning method with a negative sampling method. In our experiments with 11 real-world tabular datasets and 8 baseline methods, we prove the efficacy of the proposed method, called CoDi.

Motivation & Objective

• To address the challenge of modeling mixed-type tabular data (continuous and discrete variables) in generative modeling, where current methods often compromise inter-variable correlations.
• To improve the fidelity and diversity of synthetic tabular data by processing continuous and discrete variables in their respective physical spaces rather than mapping them to a shared continuous space.
• To enhance the alignment between continuous and discrete diffusion models through co-evolving conditional training and contrastive learning with structured negative sampling.
• To reduce training instability and improve convergence speed compared to existing approaches that rely on joint modeling of mixed types in a single diffusion process.
• To achieve a balanced solution to the generative learning trilemma (quality, diversity, and generalization) in tabular data synthesis.

Proposed method

• Two separate diffusion models are trained: one in continuous space for continuous variables and one in discrete space for discrete variables, enabling more accurate distribution learning for each type.
• The models co-evolve by conditioning each other’s forward and reverse processes at every time step, ensuring that both types are updated in a way that preserves cross-variable dependencies.
• A contrastive learning objective is applied separately to each model, using an anchor (original sample), a positive sample (generated with correct condition), and a negative sample (generated with permuted condition) to align representations.
• Negative sampling is implemented by randomly permuting the discrete condition when generating a negative continuous sample, ensuring the model learns to reject incorrect correlations.
• The contrastive loss is optimized jointly during training, reinforcing the alignment between the two models and improving generalization.
• The framework is trained end-to-end with no pre/post-processing for discrete variables beyond one-hot encoding, avoiding suboptimal sampling in continuous space.

Experimental results

Research questions

• RQ1 Can separate diffusion models for continuous and discrete tabular variables improve generation quality compared to joint modeling in a single space?
• RQ2 Does co-evolving conditional training between continuous and discrete diffusion models preserve inter-variable correlations, especially between mixed-type columns?
• RQ3 Can contrastive learning with structured negative sampling improve the alignment between discrete and continuous diffusion models and enhance sample diversity?
• RQ4 How does the proposed method compare in training stability and convergence speed to existing state-of-the-art tabular synthesis models?
• RQ5 Does the method achieve a better balance in the generative learning trilemma (quality, diversity, and generalization) than prior approaches?

Key findings

• CoDi outperforms 8 baseline methods across 11 real-world tabular datasets in both sampling quality and diversity, with significant improvements in F1 score and R² on classification and regression tasks.
• On the Bank dataset, CoDi achieved an F1 score of 0.566 ± 0.014, a 7.4% improvement over the baseline without contrastive learning (0.527 ± 0.032).
• On the Heart dataset, CoDi improved sampling diversity by 7.8% compared to the baseline, with a coverage score of 0.949 ± 0.012.
• The contrastive learning component significantly stabilized training, as shown by smooth loss curves in the Faults dataset, unlike the fluctuating curves seen in alternative negative sampling methods.
• CoDi reduced sampling time by approximately 90% compared to STaSy, the previous SOTA, due to reduced training difficulty from better model alignment.
• In the ablation study, removing contrastive learning led to degraded performance on 7 out of 11 datasets, confirming its critical role in improving both quality and diversity.