[Paper Review] Contextual Semantic Parsing for Multilingual Task-Oriented Dialogues

Mehrad Moradshahi, Victoria Tsai+2 more|arXiv (Cornell University)|Nov 4, 2021

Speech and dialogue systems4 citations

TL;DR

This paper proposes a multilingual dialogue state tracking framework that uses neural machine translation with slot-value alignment to automatically create high-quality, low-resource language datasets without human annotation. By combining a contextual semantic parsing model (BART-CSP) that encodes only the last utterances and the current belief state, it reduces translation error compounding and achieves state-of-the-art joint goal accuracy improvements of 11–52.5% on multilingual benchmarks including RiSAWOZ, CrossWOZ, and MultiWOZ-ZH.

ABSTRACT

Robust state tracking for task-oriented dialogue systems currently remains restricted to a few popular languages. This paper shows that given a large-scale dialogue data set in one language, we can automatically produce an effective semantic parser for other languages using machine translation. We propose automatic translation of dialogue datasets with alignment to ensure faithful translation of slot values and eliminate costly human supervision used in previous benchmarks. We also propose a new contextual semantic parsing model, which encodes the formal slots and values, and only the last agent and user utterances. We show that the succinct representation reduces the compounding effect of translation errors, without harming the accuracy in practice. We evaluate our approach on several dialogue state tracking benchmarks. On RiSAWOZ, CrossWOZ, CrossWOZ-EN, and MultiWOZ-ZH datasets we improve the state of the art by 11%, 17%, 20%, and 0.3% in joint goal accuracy. We present a comprehensive error analysis for all three datasets showing erroneous annotations can lead to misguided judgments on the quality of the model. Finally, we present RiSAWOZ English and German datasets, created using our translation methodology. On these datasets, accuracy is within 11% of the original showing that high-accuracy multilingual dialogue datasets are possible without relying on expensive human annotations. We release our datasets and software open source.

Motivation & Objective

• To enable high-accuracy dialogue state tracking in low-resource languages without costly human-annotated datasets.
• To address the compounding effect of translation errors in multilingual dialogue state tracking by limiting context to only the most recent utterances and current belief state.
• To develop a scalable, automated pipeline for creating multilingual dialogue datasets using neural machine translation with slot-value alignment.
• To evaluate the robustness of the proposed method on multilingual benchmarks and analyze annotation errors that may mislead model evaluation.
• To release open-source tools and datasets (RiSAWOZ-EN and RiSAWOZ-DE) for future research in multilingual task-oriented dialogues.

Proposed method

• Automatically translate existing dialogue datasets (e.g., RiSAWOZ) into target languages using neural machine translation, with a novel alignment mechanism to preserve slot-value fidelity.
• Apply a slot-value alignment model that maps source-language slots and values to their correct translations in the target language, reducing mistranslations of key entities.
• Design a Contextual Semantic Parsing (CSP) model based on BART that encodes only the current belief state and the last user and agent utterances, omitting the full dialogue history.
• Use a sequence-to-sequence transformer architecture (BART) fine-tuned on the translated datasets to predict slot-value pairs, leveraging pretrained language understanding for better generalization.
• Apply a loss function that optimizes for joint goal accuracy, focusing on correct prediction of all slot-value pairs simultaneously.
• Evaluate the model on multilingual benchmarks including RiSAWOZ, CrossWOZ, MultiWOZ-ZH, and newly created RiSAWOZ-EN-auto and RiSAWOZ-DE-auto datasets.

Experimental results

Research questions

• RQ1 Can automatic machine translation with slot-value alignment produce high-quality multilingual dialogue datasets without human post-editing?
• RQ2 Does limiting the input context to only the last utterances and current belief state reduce the compounding of translation errors in dialogue state tracking?
• RQ3 Can a contextual semantic parsing model outperform full-history models in multilingual settings, especially when training data is automatically translated?
• RQ4 How do annotation errors in existing benchmarks affect the evaluation of dialogue state tracking models?
• RQ5 To what extent can a multilingual dialogue system trained on automatically translated data achieve performance comparable to human-annotated baselines?

Key findings

• The proposed alignment method improved joint goal accuracy on RiSAWOZ by 45.6% compared to non-aligned translation, eliminating the need for costly human post-editing.
• The BART-CSP model achieved a 10.7% improvement in joint goal accuracy on RiSAWOZ and a 17% improvement on CrossWOZ, outperforming prior state-of-the-art models.
• On automatically translated datasets (RiSAWOZ-EN-auto and RiSAWOZ-DE-auto), BART-CSP achieved 32.4% and 52.5% improvements in joint goal accuracy, respectively, demonstrating strong robustness to translation noise.
• The model’s performance on the newly created RiSAWOZ-EN and RiSAWOZ-DE datasets was within 11% of the original English dataset’s performance, showing high fidelity in zero-shot multilingual transfer.
• Error analysis revealed that 33% of misannotations in CrossWOZ were missing the "Best for crowd" slot, indicating that annotation inconsistencies can mislead model evaluation.
• The model struggled with logical reasoning tasks such as price range expansion (e.g., expanding 100–150 to 80–150), often failing to infer new ranges even when the agent’s response implied it.