[Paper Review] DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Dataset

Youngjun Lee, Byungsoo Ko+3 more|arXiv (Cornell University)|Dec 8, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper presents DialogCC, a large-scale multi-modal dialogue dataset created via an automated pipeline that matches dialogues to images using CLIP-based similarity. The dataset significantly improves model generalization and achieves state-of-the-art performance in image and text retrieval, with diverse, multi-image dialogues enhancing robustness to image and text perturbations.

ABSTRACT

As sharing images in an instant message is a crucial factor, there has been active research on learning an image-text multi-modal dialogue models. However, training a well-generalized multi-modal dialogue model remains challenging due to the low quality and limited diversity of images per dialogue in existing multi-modal dialogue datasets. In this paper, we propose an automated pipeline to construct a multi-modal dialogue dataset, ensuring both dialogue quality and image diversity without requiring minimum human effort. In our pipeline, to guarantee the coherence between images and dialogue, we prompt GPT-4 to infer potential image-sharing moments - specifically, the utterance, speaker, rationale, and image description. Furthermore, we leverage CLIP similarity to maintain consistency between aligned multiple images to the utterance. Through this pipeline, we introduce DialogCC, a high-quality and diverse multi-modal dialogue dataset that surpasses existing datasets in terms of quality and diversity in human evaluation. Our comprehensive experiments highlight that when multi-modal dialogue models are trained using our dataset, their generalization performance on unseen dialogue datasets is significantly enhanced. We make our source code and dataset publicly available.

Motivation & Objective

• To address the limitations of existing multi-modal dialogue datasets, which suffer from small scale, limited topic diversity, and low image variety per dialogue.
• To develop an automated, scalable pipeline for creating high-quality multi-modal dialogue datasets without human annotation.
• To improve model generalization by enabling training on multiple diverse images per dialogue utterance.
• To demonstrate that models trained on DialogCC achieve superior performance and robustness on image and text retrieval benchmarks.

Proposed method

• The pipeline uses CLIP embeddings to automatically match utterances from text-only dialogues to relevant images from the CC3M dataset based on semantic similarity.
• It applies two filtering stages: source data filtering to remove low-quality dialogues and multi-modal dialogue filtering to ensure image-dialogue relevance.
• The method leverages pre-trained CLIP encoders for both text and image features, enabling zero-shot matching without human-in-the-loop.
• The final dataset, DialogCC, contains over 1 million dialogues with an average of 5.5 images per dialogue, covering diverse topics and conversational skills.
• The pipeline ensures high diversity by selecting images that match the semantic content of dialogues across multiple modalities.
• The resulting dataset is released publicly to support research in open-domain, multi-modal dialogue systems.

Experimental results

Research questions

• RQ1 Can an automated pipeline based on CLIP similarity effectively create a large-scale, diverse, and high-quality multi-modal dialogue dataset?
• RQ2 Does training on a dataset with multiple images per dialogue improve model generalization and robustness compared to datasets with single or few images per dialogue?
• RQ3 Can models fine-tuned on DialogCC achieve state-of-the-art performance in image and text retrieval tasks?
• RQ4 How does the model trained on DialogCC perform under image and text augmentation, indicating robustness to input variations?
• RQ5 To what extent does DialogCC mitigate the risk of memorization by reducing reliance on fixed image-utterance pairs?

Key findings

• DialogCC contains over 1 million dialogues and more than 5.5 million image-dialogue pairs, significantly outperforming existing datasets in scale and diversity.
• The dataset covers 1.5x more unique words and 2.5x more unique hypernyms than MMDD and PhotoChat, indicating superior topic and domain diversity.
• Models trained on DialogCC achieve state-of-the-art performance on both image and text retrieval benchmarks, with a 7.58% Recall@1 on the PhotoChat dataset.
• The model trained on DialogCC shows superior robustness to image augmentation, with only a 1.85% drop in Recall@1 under synonym-based text augmentation, compared to a 2.13% drop for the baseline.
• Increasing the maximum number of images per utterance from 1 to 10 improves R@1 by 1.08 points and R@10 by 1.75 points, demonstrating the benefit of multi-image training.
• The model trained on DialogCC exhibits lower performance degradation under image distortions (e.g., shearing, blurring), indicating enhanced robustness to input variations.