[Paper Review] Denoising Diffusion Probabilistic Models for Generation of Realistic Fully-Annotated Microscopy Image Data Sets

Dennis Eschweiler, Yilmaz, Rüveyda+6 more|arXiv (Cornell University)|Jan 2, 2023

Cell Image Analysis Techniques4 citations

TL;DR

This paper proposes a diffusion model-based pipeline that generates realistic, fully-annotated microscopy image datasets using only rough sketches as input, enabling fully automated, unsupervised training of segmentation models. By initiating the denoising process early and using sketch-based priors, the method produces high-fidelity image-mask pairs that train state-of-the-art segmentation models with only 200 synthetic samples, achieving performance comparable to models trained on large human-annotated datasets.

ABSTRACT

Recent advances in computer vision have led to significant progress in the generation of realistic image data, with denoising diffusion probabilistic models proving to be a particularly effective method. In this study, we demonstrate that diffusion models can effectively generate fully-annotated microscopy image data sets through an unsupervised and intuitive approach, using rough sketches of desired structures as the starting point. The proposed pipeline helps to reduce the reliance on manual annotations when training deep learning-based segmentation approaches and enables the segmentation of diverse datasets without the need for human annotations. This approach holds great promise in streamlining the data generation process and enabling a more efficient and scalable training of segmentation models, as we show in the example of different practical experiments involving various organisms and cell types.

Motivation & Objective

• To address the bottleneck of manual annotation in training deep learning models for biomedical image segmentation.
• To enable the generation of diverse, realistic, fully-annotated microscopy image datasets without requiring human-annotated data.
• To develop a scalable, unsupervised pipeline for creating synthetic training data that preserves structural fidelity and annotation accuracy.
• To evaluate whether synthetic data generated via diffusion models can train segmentation models that match performance of those trained on real annotated data.
• To explore the feasibility of replacing human annotation with simulation-based sketch generation in microscopy image synthesis.

Proposed method

• A denoising diffusion probabilistic model (DDPM) is trained on real microscopy image data to learn the backward denoising process.
• The forward process is applied to rough sketches of cellular structures to generate noisy latent states, starting from a controlled noise level at t_start < T.
• The trained backward process generates realistic image data from these sketch-derived noisy states, preserving structural information.
• The method uses PSNR and ZNCC to evaluate realism of generated images across different t_start and sketch blurring parameters.
• Segmentation models are trained from scratch on the synthetic data and evaluated on real microscopy images to assess usability.
• The pipeline leverages existing segmentation models like Cellpose or SAM to generate initial sketches, enabling full automation.

Experimental results

Research questions

• RQ1 Can denoising diffusion models generate fully-annotated microscopy image datasets from rough sketches without human-annotated training data?
• RQ2 How does the choice of t_start in the forward process affect the realism and structural fidelity of generated images?
• RQ3 Can segmentation models trained exclusively on synthetic data achieve performance comparable to those trained on large real-annotated datasets?
• RQ4 To what extent does the quality of sketch-based priors influence the realism and utility of generated image-mask pairs?
• RQ5 Can this pipeline be generalized across diverse organisms and cell types without requiring dataset-specific retraining?

Key findings

• Models trained on just 200 synthetic image-mask pairs generated via the proposed pipeline achieved segmentation accuracy comparable to state-of-the-art models trained on large real-annotated datasets.
• The Wilcoxon rank-sum test confirmed no statistically significant difference in segmentation performance between models trained on synthetic data and those fine-tuned on real data, with p-values as low as 0.0002.
• High PSNR and ZNCC values were consistently achieved across all datasets, indicating strong realism in generated image data.
• The method successfully generated realistic image data for five diverse biological datasets, including A. thaliana, C. elegans, T. castaneum, mouse stem cells, and HeLa cells.
• The approach enables full automation of data generation, eliminating the need for human annotations while maintaining high-quality segmentation performance.
• Despite limitations in generating very dark cells due to noise-induced structural loss, the method remains effective for most practical segmentation tasks.