[Paper Review] Are Current Task-oriented Dialogue Systems Able to Satisfy Impolite Users?

Zhiqiang Hu, Roy Kaa-Wei Lee+1 more|arXiv (Cornell University)|Oct 24, 2022

AI in Service Interactions4 citations

TL;DR

This paper investigates the robustness of task-oriented dialogue (TOD) systems when interacting with impolite users, constructing a large-scale impolite dialogue corpus by rewriting 10,000 utterances from MultiWOZ 2.2. Experiments show existing TOD systems fail to handle impolite users effectively, but a text style transfer-based data augmentation method improves performance, highlighting the need for more robust, impolite-user-aware dialogue systems.

ABSTRACT

Task-oriented dialogue (TOD) systems have assisted users on many tasks, including ticket booking and service inquiries. While existing TOD systems have shown promising performance in serving customer needs, these systems mostly assume that users would interact with the dialogue agent politely. This assumption is unrealistic as impatient or frustrated customers may also interact with TOD systems impolitely. This paper aims to address this research gap by investigating impolite users' effects on TOD systems. Specifically, we constructed an impolite dialogue corpus and conducted extensive experiments to evaluate the state-of-the-art TOD systems on our impolite dialogue corpus. Our experimental results show that existing TOD systems are unable to handle impolite user utterances. We also present a data augmentation method to improve TOD performance in impolite dialogues. Nevertheless, handling impolite dialogues remains a very challenging research task. We hope by releasing the impolite dialogue corpus and establishing the benchmark evaluations, more researchers are encouraged to investigate this new challenging research task.

Motivation & Objective

• To address the lack of research on how task-oriented dialogue systems handle impolite users, especially in high-stress or frustrated scenarios.
• To investigate the performance degradation of state-of-the-art TOD systems when faced with impolite user utterances.
• To build a realistic, diverse, and large-scale impolite dialogue corpus to enable benchmark evaluation of TOD systems under impolite conditions.
• To explore data augmentation techniques that can improve TOD system robustness in handling impolite user inputs.
• To encourage future research on building more resilient TOD systems capable of maintaining performance under user impoliteness.

Proposed method

• Constructed an impolite dialogue corpus by manually rewriting 10,000 user utterances from the MultiWOZ 2.2 dataset using role-playing scenarios to induce impolite behavior.
• Employed human annotators to rewrite utterances under specific emotional states (e.g., frustration, urgency) to generate diverse and naturalistic impolite expressions.
• Performed statistical and linguistic analysis on the corpus, including politeness scoring and semantic similarity computation using sentence embeddings.
• Evaluated six state-of-the-art TOD systems (e.g., PPTOD) on the impolite corpus using standard automatic metrics (e.g., joint goal accuracy, slot F1).
• Proposed a data augmentation method using text style transfer to generate synthetic impolite utterances from polite training data.
• Fine-tuned TOD models on the augmented data to assess improvements in handling impolite user inputs.

Experimental results

Research questions

• RQ1 How do existing state-of-the-art task-oriented dialogue systems perform when interacting with impolite users?
• RQ2 What are the linguistic and semantic characteristics of impolite user utterances that challenge current TOD systems?
• RQ3 Does the politeness level of user utterances correlate with the performance drop in TOD systems?
• RQ4 Can data augmentation via text style transfer improve the robustness of TOD systems toward impolite user inputs?
• RQ5 How does semantic similarity between original and rewritten utterances affect system performance in impolite dialogue scenarios?

Key findings

• Existing state-of-the-art TOD systems, including PPTOD, show significant performance degradation when handling impolite user utterances, with joint goal accuracy dropping substantially.
• The performance drop is most severe not for the most abusive utterances, but for moderately impolite, acerbic expressions that are less overtly offensive but more linguistically complex.
• Systems struggle most with impolite dialogues that have low semantic similarity to their original counterparts, indicating difficulty generalizing beyond paraphrased or restructured utterances.
• The proposed text style transfer-based data augmentation method improves TOD system performance on impolite dialogues, demonstrating its potential for enhancing robustness.
• The constructed impolite dialogue corpus reveals diverse impolite expression patterns, including sarcasm, urgency, and emotional outbursts, which current systems fail to interpret correctly.
• Even when users are extremely impolite (e.g., using abusive language), some systems still provide partially correct responses, suggesting that extreme rudeness is less disruptive than subtle, sharp impoliteness.