[Paper Review] Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation

Olga Majewska, Evgeniia Razumovskaia+3 more|arXiv (Cornell University)|Jan 31, 2022

Topic Modeling5 citations

TL;DR

This paper proposes an outline-based method for creating high-quality, multilingual task-oriented dialogue datasets by transforming domain-specific dialogue schemata into natural language outlines that guide native speakers in generating culturally appropriate, natural-sounding dialogues in multiple languages. The resulting Cross-lingual Outline-based Dialogue (cod) dataset enables robust evaluation of multilingual and cross-lingual dialogue systems across 11 domains and 4 diverse languages, demonstrating superior data quality and more realistic performance benchmarks than translation-based methods.

ABSTRACT

Multilingual task-oriented dialogue (ToD) facilitates access to services and information for many (communities of) speakers. Nevertheless, the potential of this technology is not fully realised, as current datasets for multilingual ToD - both for modular and end-to-end modelling - suffer from severe limitations. 1) When created from scratch, they are usually small in scale and fail to cover many possible dialogue flows. 2) Translation-based ToD datasets might lack naturalness and cultural specificity in the target language. In this work, to tackle these limitations we propose a novel outline-based annotation process for multilingual ToD datasets, where domain-specific abstract schemata of dialogue are mapped into natural language outlines. These in turn guide the target language annotators in writing a dialogue by providing instructions about each turn's intents and slots. Through this process we annotate a new large-scale dataset for training and evaluation of multilingual and cross-lingual ToD systems. Our Cross-lingual Outline-based Dialogue dataset (termed COD) enables natural language understanding, dialogue state tracking, and end-to-end dialogue modelling and evaluation in 4 diverse languages: Arabic, Indonesian, Russian, and Kiswahili. Qualitative and quantitative analyses of COD versus an equivalent translation-based dataset demonstrate improvements in data quality, unlocked by the outline-based approach. Finally, we benchmark a series of state-of-the-art systems for cross-lingual ToD, setting reference scores for future work and demonstrating that COD prevents over-inflated performance, typically met with prior translation-based ToD datasets.

Motivation & Objective

• To address the limitations of existing multilingual task-oriented dialogue (ToD) datasets, which are often small, lack coverage of dialogue flows, or suffer from unnaturalness and cultural inappropriateness due to translation-based creation.
• To develop a scalable, cost-effective method for creating high-quality multilingual ToD data that preserves linguistic and cultural authenticity while enabling cross-lingual transfer learning.
• To create a large-scale, multilingual, multi-domain dialogue dataset suitable for training and evaluating end-to-end, dialogue state tracking, and natural language understanding models across diverse languages.
• To provide a realistic benchmark that prevents over-optimistic performance estimates common in translation-based datasets, thereby enabling more reliable evaluation of cross-lingual ToD systems.

Proposed method

• Domain-specific, language-independent dialogue schemata are extracted from the English Schema-Guided Dialogue (SGD) dataset as abstract templates.
• These schemata are automatically mapped into natural language outlines in English, specifying the intent and required slot values for each dialogue turn.
• Native speakers in target languages (Arabic, Indonesian, Russian, Kiswahili) use these outlines as instructions to generate natural, culturally adapted dialogues, replacing slot values with local context.
• The process ensures cross-lingual comparability while preserving linguistic naturalness and cultural specificity, avoiding the pitfalls of direct translation.
• The final dataset, cod, supports multiple tasks: intent detection, slot labelling, dialogue state tracking, and end-to-end dialogue generation across 11 domains and 4 languages.
• Ablation studies compare cod to a translation-based baseline, demonstrating qualitative and quantitative improvements in data quality and model generalization.

Experimental results

Research questions

• RQ1 Can an outline-based annotation pipeline produce multilingual task-oriented dialogue data that is more natural and culturally appropriate than translation-based methods?
• RQ2 Does the proposed method enable better generalization and more realistic performance evaluation for cross-lingual task-oriented dialogue systems?
• RQ3 To what extent does the cod dataset support evaluation across multiple domains and languages, particularly in cross-domain and end-to-end settings?
• RQ4 How do state-of-the-art cross-lingual models perform on cod compared to translation-based benchmarks, and does cod prevent over-inflated performance estimates?

Key findings

• The outline-based method produces dialogues with significantly higher naturalness and cultural specificity compared to translation-based baselines, as confirmed by qualitative analysis.
• The cod dataset enables more realistic evaluation of cross-lingual transfer, preventing the over-optimistic performance typically seen in translation-based benchmarks.
• In-domain intent detection accuracy on cod reaches 32.07% (XLM-R) and 53.36% F1 for slot labelling (XLM-R), with substantial room for improvement, indicating the dataset's challenge and utility.
• Cross-domain performance is substantially lower than in-domain, highlighting the difficulty of zero-shot or few-shot cross-domain transfer and validating the dataset’s suitability for such evaluations.
• BLEU scores for end-to-end generation are low (e.g., 3.76 for mT5 in cross-domain), confirming that the dataset poses a meaningful challenge for end-to-end models.
• The cod dataset is the first multilingual ToD resource that supports evaluation across 11 domains and 4 typologically diverse languages, enabling comprehensive benchmarking of NLU, DST, and E2E systems.