[Paper Review] DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI

Jianguo Zhang, Kun Qian+8 more|arXiv (Cornell University)|Jul 19, 2023

Topic ModelingComputer Science3 citations

TL;DR

DialogStudio introduces the largest and most diverse unified collection of 80+ dialogue datasets spanning open-domain, task-oriented, knowledge-grounded, and summarization tasks, standardized into a consistent format with licenses, domain-aware prompts, and external knowledge. Trained models using DialogStudio achieve state-of-the-art performance in zero-shot and few-shot settings, demonstrating its superiority for conversational AI research and training.

ABSTRACT

Despite advancements in conversational AI, language models encounter challenges to handle diverse conversational tasks, and existing dialogue dataset collections often lack diversity and comprehensiveness. To tackle these issues, we introduce DialogStudio: the largest and most diverse collection of dialogue datasets, unified under a consistent format while preserving their original information. Our collection encompasses data from open-domain dialogues, task-oriented dialogues, natural language understanding, conversational recommendation, dialogue summarization, and knowledge-grounded dialogues, making it an incredibly rich and diverse resource for dialogue research and model training. To further enhance the utility of DialogStudio, we identify the licenses for each dataset, design external knowledge and domain-aware prompts for selected dialogues to facilitate instruction-aware fine-tuning. Furthermore, we develop conversational AI models using the dataset collection, and our experiments in both zero-shot and few-shot learning scenarios demonstrate the superiority of DialogStudio. To improve transparency and support dataset and task-based research, as well as language model pre-training, all datasets, licenses, codes, and models associated with DialogStudio are made publicly accessible\footnote{\url{https://github.com/salesforce/DialogStudio}}.

Motivation & Objective

• To address the lack of comprehensive, diverse, and standardized dialogue datasets that hinder model generalization in conversational AI.
• To unify disparate dialogue datasets from multiple domains and tasks into a single, consistent, and accessible format while preserving original information.
• To enhance dataset usability by identifying licenses, designing domain-aware prompts, and integrating external knowledge for instruction tuning.
• To support holistic research and pre-training by enabling task-specific and dataset-based analysis across diverse conversational scenarios.
• To improve model performance through zero-shot and few-shot learning by leveraging the rich, unified data distribution in DialogStudio.

Proposed method

• Aggregating over 80 dialogue datasets from diverse sources across six categories: open-domain, task-oriented, NLU, conversational recommendation, dialogue summarization, and knowledge-grounded dialogues.
• Standardizing all datasets into a consistent JSON format with unified metadata, while preserving original structure and content.
• Identifying and documenting licenses for each dataset to ensure legal compliance and transparency in research.
• Designing domain-aware and instruction-tuning prompts for selected datasets to facilitate fine-tuning of instruction-following models.
• Creating external knowledge injections for knowledge-grounded dialogues to improve model reasoning and response generation.
• Training instruction-aware conversational models (770M to 3B parameters) using the DialogStudio collection for zero-shot and few-shot evaluation.

Experimental results

Research questions

• RQ1 Can a unified, diverse, and standardized dialogue dataset collection improve zero-shot and few-shot performance in conversational AI models?
• RQ2 To what extent does including multiple dialogue tasks and domains in a single collection enhance model generalization and robustness?
• RQ3 How effective is the integration of domain-aware prompts and external knowledge in improving model instruction-following and response quality?
• RQ4 Can a unified dataset collection support both task-specific research and large-scale language model pre-training effectively?
• RQ5 How does DialogStudio compare to existing collections like Flan, OPT, or ParlAI in terms of coverage, diversity, and usability?

Key findings

• DialogStudio includes over 80 dialogue datasets across six distinct categories, making it the most comprehensive and diverse unified collection to date.
• The dataset collection is publicly available with full licensing information, ensuring transparency and reproducibility in research.
• The use of domain-aware prompts and external knowledge significantly enhances model performance in instruction-following and knowledge-grounded tasks.
• Instruction-tuned models trained on DialogStudio outperform strong baselines in both zero-shot and few-shot learning scenarios.
• The unified HuggingFace-compatible interface with a simple `load_dataset()` command enables seamless integration and broad usability across research and development workflows.
• The collection supports both individual dataset analysis and large-scale pre-training, enabling fair comparisons and scalable model development.