[Paper Review] Adding Chit-Chats to Enhance Task-Oriented Dialogues

Kai Sun, Seungwhan Moon+8 more|arXiv (Cornell University)|Oct 24, 2020

Topic Modeling42 references4 citations

TL;DR

This paper proposes ACCENTOR, a framework that enhances task-oriented dialogues by integrating human-in-the-loop collected chit-chat responses, enabling virtual assistants to switch naturally between task execution and socially engaging conversation. Evaluated on Schema-Guided Dialogue and MultiWOZ 2.1, the approach improves engagement and humanlikeness without sacrificing task performance.

ABSTRACT

Existing dialogue corpora and models are typically designed under two disjoint motives: while task-oriented systems focus on achieving functional goals (e.g., booking hotels), open-domain chatbots aim at making socially engaging conversations. In this work, we propose to integrate both types of systems by Adding Chit-Chat to ENhance Task-ORiented dialogues (ACCENTOR), with the goal of making virtual assistant conversations more engaging and interactive. Specifically, we propose a Human AI collaborative data collection approach for generating diverse chit-chat responses to augment task-oriented dialogues with minimal annotation effort. We then present our new chit-chat-based annotations to 23.8K dialogues from two popular task-oriented datasets (Schema-Guided Dialogue and MultiWOZ 2.1) and demonstrate their advantage over the originals via human evaluation. Lastly, we propose three new models for adding chit-chat to task-oriented dialogues, explicitly trained to predict user goals and to generate contextually relevant chit-chat responses. Automatic and human evaluations show that, compared with the state-of-the-art task-oriented baseline, our models can code-switch between task and chit-chat to be more engaging, interesting, knowledgeable, and humanlike, while maintaining competitive task performance.

Motivation & Objective

• To bridge the gap between task-oriented dialogue systems and open-domain chatbots by integrating chit-chat into functional dialogues.
• To reduce annotation effort in collecting diverse chit-chat responses through a human-AI collaborative data collection method.
• To improve the naturalness, engagement, and humanlikeness of virtual assistant conversations while preserving task success rates.
• To develop models capable of code-switching between task-oriented and chit-chat responses based on user goals and context.

Proposed method

• A human-AI collaborative data collection pipeline was used to generate diverse chit-chat responses to task-oriented dialogue turns with minimal human annotation effort.
• The collected chit-chat responses were annotated on 23.8K dialogues from Schema-Guided Dialogue and MultiWOZ 2.1, enriching the original datasets.
• Three new neural models were trained to jointly predict user goals and generate contextually relevant chit-chat responses.
• The models are trained end-to-end to enable dynamic switching between task-oriented and chit-chat modes based on dialogue context.
• Human evaluation was used to validate the quality and relevance of the generated chit-chat responses.
• Automatic and human evaluations were conducted to compare the proposed models against state-of-the-art task-oriented baselines.

Experimental results

Research questions

• RQ1 Can chit-chat responses improve the engagement and humanlikeness of task-oriented dialogues without degrading task performance?
• RQ2 How can chit-chat be collected efficiently at scale while preserving diversity and relevance?
• RQ3 To what extent can models learn to code-switch between task and chit-chat modes based on context and user goals?
• RQ4 How do human-annotated chit-chat responses compare to baseline models in terms of quality and naturalness?
• RQ5 Can end-to-end models trained on joint goal prediction and chit-chat generation outperform separate task and response generation systems?

Key findings

• Human evaluation confirmed that dialogues enhanced with chit-chat were rated significantly more engaging, interesting, and humanlike compared to original task-oriented dialogues.
• The proposed models achieved competitive task performance, maintaining high success rates on both Schema-Guided Dialogue and MultiWOZ 2.1 despite adding chit-chat components.
• The human-AI collaborative data collection method reduced annotation effort while generating diverse and contextually relevant chit-chat responses.
• Models trained with joint goal prediction and chit-chat generation demonstrated improved code-switching behavior between task and social modes.
• Automatic evaluations showed that the models generated more contextually relevant and diverse chit-chat responses than baseline systems.