[Paper Review] Cap2Aug: Caption guided Image to Image data Augmentation

Aniket Roy, Anshul Shah+3 more|arXiv (Cornell University)|Dec 11, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

Cap2Aug proposes a novel image-to-image data augmentation method that uses image captions as text prompts to generate semantically diverse, realistic training samples via a stable diffusion model. By combining caption-guided image generation with a maximum mean discrepancy (MMD) loss to reduce domain shift, the method achieves state-of-the-art performance in few-shot and long-tail image classification, especially under low-data regimes.

ABSTRACT

Visual recognition in a low-data regime is challenging and often prone to overfitting. To mitigate this issue, several data augmentation strategies have been proposed. However, standard transformations, e.g., rotation, cropping, and flipping provide limited semantic variations. To this end, we propose Cap2Aug, an image-to-image diffusion model-based data augmentation strategy using image captions as text prompts. We generate captions from the limited training images and using these captions edit the training images using an image-to-image stable diffusion model to generate semantically meaningful augmentations. This strategy generates augmented versions of images similar to the training images yet provides semantic diversity across the samples. We show that the variations within the class can be captured by the captions and then translated to generate diverse samples using the image-to-image diffusion model guided by the captions. However, naive learning on synthetic images is not adequate due to the domain gap between real and synthetic images. Thus, we employ a maximum mean discrepancy (MMD) loss to align the synthetic images to the real images for minimizing the domain gap. We evaluate our method on few-shot and long-tail classification tasks and obtain performance improvements over state-of-the-art, especially in the low-data regimes.

Motivation & Objective

• To address the challenge of overfitting in low-data regimes by generating semantically meaningful data augmentations beyond standard geometric transformations.
• To leverage pre-trained vision-language models and diffusion models to create diverse, realistic image variations guided by natural language descriptions.
• To reduce the domain gap between real and synthetic images by aligning their feature distributions using a maximum mean discrepancy (MMD) loss.
• To improve few-shot and long-tail classification performance through semantically rich, caption-guided image generation.
• To validate the method on standard benchmarks and analyze failure cases, particularly in fine-grained recognition.

Proposed method

• Generate captions for training images using a pre-trained image captioning model to capture semantic attributes.
• Use an image-to-image stable diffusion model to generate augmented images by conditioning on both the original image and its caption as a text prompt.
• Enable cross-caption generation by using captions from one image to augment a different image, increasing inter-class semantic diversity.
• Apply a multi-kernel maximum mean discrepancy (MMD) loss to align the feature distributions of synthetic images with those of real images, minimizing domain shift.
• Fine-tune the classifier on the combined real and augmented dataset to improve generalization.
• Use ablation studies to validate the contribution of captioning, image-to-image generation, and MMD loss components.

Experimental results

Research questions

• RQ1 Can caption-guided image-to-image generation produce more semantically diverse and realistic data augmentations than standard geometric augmentations?
• RQ2 How effective is the MMD-based domain alignment in reducing the performance gap between real and synthetic images?
• RQ3 Does Cap2Aug improve few-shot and long-tail classification accuracy compared to state-of-the-art methods?
• RQ4 In what scenarios does the method fail, particularly in fine-grained recognition tasks?
• RQ5 Can cross-caption generation (using captions from one image to edit another) significantly improve model generalization?

Key findings

• Cap2Aug achieves state-of-the-art performance on 11 few-shot learning benchmarks, with a mean accuracy of 92.80% on the 5-way 5-shot setting, outperforming prior methods.
• On the ImageNet-LT benchmark for long-tail classification, Cap2Aug improves over existing SOTA methods, demonstrating effectiveness in long-tail data regimes.
• The MMD-based domain alignment significantly reduces the domain gap, improving generalization when combining real and synthetic images.
• Ablation studies confirm that both caption-guided image generation and MMD loss are essential components for performance gains.
• Failure cases occur in fine-grained classification tasks such as FGVC-Aircraft and Food101, where captions fail to capture critical visual distinctions, leading to misleading augmentations.
• The method is not effective for 1-shot learning due to insufficient image diversity, and thus is evaluated from 2-shot onwards.