[Paper Review] Model-based Cleaning of the QUILT-1M Pathology Dataset for Text-Conditional Image Synthesis

Marc Aubreville, Jonathan Ganz+3 more|arXiv (Cornell University)|Apr 11, 2024

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes a model-based pipeline to clean the QUILT-1M histopathology dataset for text-conditional image synthesis by detecting and removing visual impurities—such as narrators, software overlays, and text annotations—using a ResNet50-D-based multi-label classifier and semantic alignment filtering via CLIP scores. The method significantly improves image fidelity in text-to-image generation, reducing artifacts and lowering FID scores on downstream diffusion models.

ABSTRACT

The QUILT-1M dataset is the first openly available dataset containing images harvested from various online sources. While it provides a huge data variety, the image quality and composition is highly heterogeneous, impacting its utility for text-conditional image synthesis. We propose an automatic pipeline that provides predictions of the most common impurities within the images, e.g., visibility of narrators, desktop environment and pathology software, or text within the image. Additionally, we propose to use semantic alignment filtering of the image-text pairs. Our findings demonstrate that by rigorously filtering the dataset, there is a substantial enhancement of image fidelity in text-to-image tasks.

Motivation & Objective

• To address the poor image quality and visual impurities in the QUILT-1M dataset, which limits its utility for text-conditional image synthesis.
• To develop an automated, scalable method for detecting common image impurities in histopathology images sourced from public online repositories.
• To improve the semantic alignment between image-text pairs in large-scale datasets using CLIP-based scoring for filtering.
• To evaluate the impact of dataset cleaning on downstream text-to-image generation quality using FID and qualitative analysis.
• To enable high-fidelity, pathology-specific image generation by filtering out low-quality and non-pure image content

Proposed method

• Trained a ResNet50-D-based multi-label classifier on a 1% manual annotation of 6,532 images from QUILT-1M to detect nine types of visual impurities, including narrators, desktop elements, and text overlays.
• Used early stopping and model selection based on validation loss to optimize the impurity detection model.
• Applied semantic alignment filtering by computing CLIP scores via the CONCH model and retaining only the top 50% of image-text pairs with the highest scores.
• Combined both filtering steps: first impurity detection, then semantic alignment, to produce a cleaned dataset subset.
• Fine-tuned a latent diffusion model (Stable Diffusion 2.1) on the filtered dataset for 15,000 iterations to evaluate text-to-image generation quality.
• Evaluated generation quality using conditional Fréchet Inception Distance (FID) on 10,000 generated image crops, comparing against baseline and benchmark datasets (MIDOG++ and PLISM).

Experimental results

Research questions

• RQ1 To what extent can a deep learning model detect common visual impurities in histopathology images from publicly scraped sources like QUILT-1M?
• RQ2 How does filtering based on semantic alignment (via CLIP scores) improve the quality of image-text pairs in a large-scale dataset?
• RQ3 What is the impact of removing impurities and improving alignment on the fidelity of text-to-image generation in histopathology?
• RQ4 Can a cleaned dataset lead to a measurable reduction in artifacts and improved FID scores in diffusion-based image generation models?

Key findings

• The multi-label impurity classifier achieved 92.71% accuracy on the test set, with high recall (97.17%) and AUC (0.9481) for detecting any impurity.
• Over 78% of images in the QUILT-1M dataset contained at least one visual impurity, with desktop/software elements (35.96%) and additional buttons/control elements (31.89%) being the most prevalent.
• Models trained on the unfiltered QUILT-1M dataset produced noticeable artifacts in generated images, as visually confirmed in qualitative samples.
• The filtered dataset led to a significant improvement in image fidelity, with a 32% reduction in FID score compared to the unfiltered baseline.
• Semantic alignment filtering via CONCH’s CLIP score effectively removed poorly aligned image-text pairs, enhancing the quality of training data.
• The combination of impurity detection and semantic filtering yielded a cleaner, more reliable dataset for training high-fidelity text-to-image models in histopathology.