[Paper Review] Class-Guided Image-to-Image Diffusion: Cell Painting from Brightfield Images with Class Labels

Jan Cross-Zamirski, Praveen Anand+4 more|arXiv (Cornell University)|Mar 15, 2023

Cell Image Analysis TechniquesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper proposes a class-guided image-to-image diffusion model that leverages discrete perturbation labels to improve cell painting image generation from brightfield microscopy inputs. By integrating class labels into a diffusion-based image-to-image framework, the method enhances morphological feature fidelity and downstream drug mechanism-of-action prediction performance, especially when trained on biologically active subsets, outperforming unguided models in feature correlation and structural similarity.

ABSTRACT

Image-to-image reconstruction problems with free or inexpensive metadata in the form of class labels appear often in biological and medical image domains. Existing text-guided or style-transfer image-to-image approaches do not translate to datasets where additional information is provided as discrete classes. We introduce and implement a model which combines image-to-image and class-guided denoising diffusion probabilistic models. We train our model on a real-world dataset of microscopy images used for drug discovery, with and without incorporating metadata labels. By exploring the properties of image-to-image diffusion with relevant labels, we show that class-guided image-to-image diffusion can improve the meaningful content of the reconstructed images and outperform the unguided model in useful downstream tasks.

Motivation & Objective

• To address the challenge of generating high-fidelity fluorescent cell painting images from low-cost brightfield microscopy inputs in drug discovery.
• To investigate whether discrete class labels (e.g., perturbation type) can guide image-to-image diffusion models to produce biologically meaningful reconstructions.
• To evaluate model performance not only via standard image metrics but also through downstream biological tasks like mechanism-of-action prediction and clustering.
• To demonstrate that label-guided diffusion can outperform unguided image-to-image models when trained on biologically relevant, active compound subsets.

Proposed method

• The method extends the Palette image-to-image diffusion framework with class-guided denoising, using a classifier guidance mechanism to condition generation on discrete perturbation labels.
• It employs a U-Net-based denoising network that takes both a brightfield input image and a class label embedding as input to predict the corresponding fluorescent channel images.
• The model is trained on a subset of the JUMP-CP Target2 dataset, using 3-channel brightfield images as input and 5-channel Cell Painting images as targets.
• Class labels are embedded via a learned projection head and used in the denoising process through classifier guidance, which steers the diffusion process toward desired phenotypic outputs.
• The framework supports both full-plate and active-subset training regimes to assess label quality and data sparsity effects.
• Performance is evaluated using image metrics (FID, SSIM), feature correlation with CellProfiler, and downstream target matching in mechanism-of-action prediction tasks.

Figure 2: Given input Brightfield (3 channels) our model is able to generate 5 Cell Painting channels. Incorporating meaningful labels can improve biological feature quality and performance on downstream tasks without significantly reducing image quality or adding background noise. Columns left to r

Experimental results

Research questions

• RQ1 Can discrete class labels improve the quality and biological relevance of image-to-image diffusion reconstructions in microscopy?
• RQ2 Does incorporating perturbation labels enhance downstream drug profiling tasks such as mechanism-of-action prediction and clustering?
• RQ3 How does the quality of training data—particularly the inclusion of inactive or noisy samples—affect the performance of class-guided diffusion models?
• RQ4 To what extent do class-guided models outperform unguided image-to-image baselines in feature fidelity and image reconstruction metrics?

Key findings

• When trained on the active subset of perturbations, the class-guided model achieved the highest correlation with ground-truth morphological features from CellProfiler, outperforming the unguided Palette model.
• The model improved structural similarity (SSIM) and target matching accuracy in mechanism-of-action prediction, even with a smaller training set, demonstrating robustness to data scarcity.
• Using perturbation labels as conditioning improved target matching by 15% compared to the unguided model, indicating better phenotypic generalization.
• The model trained on the full plate with perturbation labels achieved the lowest FID score (14.2), though it introduced some background noise, suggesting a trade-off between image fidelity and biological relevance.
• Classifier guidance improved target matching but slightly reduced image and feature quality, indicating a need for careful label selection.
• The results show that high-quality, biologically informative labels are critical—poor or uninformative labels can degrade image quality and mislead the model.

Figure 3: Images generated by models trained with the active subset. The labelled images were sampled with both AdaGN and CG. Cropped to $100\times 100$ pixels. Columns left to right: Brightfield (input), DNA, RNA, ER, Mito, AGP.