[Paper Review] MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentation

Jiahao Xie, Wei Li+4 more|arXiv (Cornell University)|Sep 22, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

MosaicFusion proposes a training-free, label-supervision-free method to generate high-quality synthetic images and instance masks for large vocabulary instance segmentation using pre-trained text-to-image diffusion models. By dividing an image canvas into regions and conditioning each on a different text prompt during a single diffusion process, it leverages aggregated cross-attention maps across layers and time steps to generate precise instance masks via thresholding and edge-aware refinement, significantly boosting performance on rare and novel categories in LVIS benchmarks.

ABSTRACT

We present MosaicFusion, a simple yet effective diffusion-based data augmentation approach for large vocabulary instance segmentation. Our method is training-free and does not rely on any label supervision. Two key designs enable us to employ an off-the-shelf text-to-image diffusion model as a useful dataset generator for object instances and mask annotations. First, we divide an image canvas into several regions and perform a single round of diffusion process to generate multiple instances simultaneously, conditioning on different text prompts. Second, we obtain corresponding instance masks by aggregating cross-attention maps associated with object prompts across layers and diffusion time steps, followed by simple thresholding and edge-aware refinement processing. Without bells and whistles, our MosaicFusion can produce a significant amount of synthetic labeled data for both rare and novel categories. Experimental results on the challenging LVIS long-tailed and open-vocabulary benchmarks demonstrate that MosaicFusion can significantly improve the performance of existing instance segmentation models, especially for rare and novel categories. Code: https://github.com/Jiahao000/MosaicFusion.

Motivation & Objective

• Address the challenge of data scarcity in large vocabulary instance segmentation, especially for rare and novel categories.
• Overcome the limitations of existing data augmentation methods that require training or label supervision.
• Leverage pre-trained text-to-image diffusion models to generate diverse, high-quality images and corresponding instance masks simultaneously.
• Enable training-free data augmentation for downstream instance segmentation models without modifying the backbone architecture.
• Improve model generalization and performance on long-tailed and open-vocabulary instance segmentation benchmarks.

Proposed method

• Divide the image canvas into multiple regions to enable simultaneous generation of multiple objects in a single image.
• Run a single diffusion process across all regions, conditioning each on a distinct text prompt to generate diverse, photorealistic object instances.
• Extract cross-attention maps corresponding to each object's text prompt across all transformer layers and diffusion time steps.
• Aggregate these attention maps by averaging across layers and time steps to produce high-quality, spatially coherent mask proposals.
• Apply thresholding and edge-aware refinement (e.g., Bilateral Solver) to the aggregated attention maps to generate precise instance masks.
• Use the generated image-mask pairs as synthetic training data to fine-tune or pre-train existing instance segmentation models.

Experimental results

Research questions

• RQ1 Can a pre-trained text-to-image diffusion model be used to generate diverse, multi-object images with accurate instance masks without additional training or supervision?
• RQ2 How effective is the aggregation of cross-attention maps across layers and time steps in producing high-quality mask predictions?
• RQ3 To what extent can synthetic data generated via MosaicFusion improve instance segmentation performance on rare and novel categories in long-tailed and open-vocabulary settings?
• RQ4 Does the method generalize across different backbone architectures and instance segmentation models?
• RQ5 What is the impact of the number of generated objects per image on the quality and utility of the synthetic dataset?

Key findings

• MosaicFusion significantly improves mask AP on rare categories (AP_r) and novel categories (AP_novel) across multiple baseline models, including Mask R-CNN, Box-Supervised CenterNet2, and F-VLM.
• The method achieves strong gains on the LVIS long-tailed and open-vocabulary benchmarks without any model fine-tuning or additional supervision.
• Aggregating cross-attention maps across layers and time steps yields higher-quality mask predictions than using individual maps, with the best results observed in the final diffusion steps and highest-resolution layers.
• The synthetic dataset generated by MosaicFusion enables state-of-the-art performance improvements, particularly for low-resource categories, demonstrating the method’s effectiveness in data-scarce scenarios.
• The approach is compatible with various downstream instance segmentation models, showing consistent gains regardless of the backbone architecture used.
• Despite a domain gap, the synthetic images and masks are sufficiently realistic and diverse to serve as effective training data, outperforming traditional augmentation techniques like Copy-Paste in long-tail generalization.